C# Class IronKonoha.Tokenizer

Mostra file Open project: tetsurom/ikonoha Class Usage Examples

Public Methods

Method	Description
Tokenize ( String script ) : IList
Tokenizer ( Context ctx, KonohaSpace ks ) : System

Private Methods

Method	Description
IsAlphaOrNum ( int c ) : bool
IsHexNumChar ( int c ) : bool
IsNumChar ( int c ) : bool
IsSymbolic ( int c ) : bool
TokenizeBlock ( Context ctx, Token &token, TokenizerEnvironment tenv, int tokStart, KMethod thunk ) : int
TokenizeComment ( Context ctx, Token &token, TokenizerEnvironment tenv, int tokStart, KMethod thunk ) : int
TokenizeDoubleQuote ( Context ctx, Token &token, TokenizerEnvironment tenv, int tokStart, KMethod thunk ) : int
TokenizeIndent ( Context ctx, Token &token, TokenizerEnvironment tenv, int tokStart, KMethod thunk ) : int
TokenizeLine ( Context ctx, Token &token, TokenizerEnvironment tenv, int tokStart, KMethod thunk ) : int
TokenizeNextline ( Context ctx, Token &token, TokenizerEnvironment tenv, int tokStart, KMethod thunk ) : int
TokenizeNumber ( Context ctx, Token &token, TokenizerEnvironment tenv, int tokStart, KMethod thunk ) : int
TokenizeOneCharOperator ( Context ctx, Token &token, TokenizerEnvironment tenv, int tokStart, KMethod thunk ) : int
TokenizeOperator ( Context ctx, Token &token, TokenizerEnvironment tenv, int tokStart, KMethod thunk ) : int
TokenizeSkip ( Context ctx, Token &token, TokenizerEnvironment tenv, int tokStart, KMethod thunk ) : int
TokenizeSlash ( Context ctx, Token &token, TokenizerEnvironment tenv, int tokStart, KMethod thunk ) : int
TokenizeSymbol ( Context ctx, Token &token, TokenizerEnvironment tenv, int tokStart, KMethod thunk ) : int
TokenizeUndefined ( Context ctx, Token &token, TokenizerEnvironment tenv, int tokStart, KMethod thunk ) : int

Method Details

Tokenize() public method

public Tokenize ( String script ) : IList
script	String
return	IList

Tokenizer() public method

public Tokenizer ( Context ctx, KonohaSpace ks ) : System
ctx	Context
ks	KonohaSpace
return	System