C# Class SimpleC.Lexing.Tokenizer

Tokenizer for the SimpleC language.
Show file Open project: nrother/simple-c Class Usage Examples

Public Methods

Method Description
Tokenize ( ) : SimpleC.Types.Token[]
Tokenizer ( string code ) : System

Private Methods

Method Description
charTypeOf ( char c ) : CharType
eof ( ) : bool

Returns true if there are no more chars left.

next ( ) : char

Returns the next character and advances the pointer.

nextType ( ) : CharType

Returns the CharType of the next char, advancing the pointer.

peek ( ) : char

Returns the next character without advancing the pointer.

peekType ( ) : CharType

Returns the CharType of the next char, without advancing the pointer.

readToken ( StringBuilder builder, CharType typeToRead ) : void

Reads characters into the StringBuilder while they match the given type(s).

skip ( CharType typeToSkip ) : void

Skips any count of occurences of the given char class(es).

Method Details

Tokenize() public method

public Tokenize ( ) : SimpleC.Types.Token[]
return SimpleC.Types.Token[]

Tokenizer() public method

public Tokenizer ( string code ) : System
code string
return System