C# Class Shovel.Compiler.Tokenizer

Show file Open project: mbrezu/Shovel Class Usage Examples

Public Methods

Method Description
Tokenizer ( SourceFile source, int initialPos, int limit = -1 ) : System

Private Methods

Method Description
CurrentChar ( ) : char
EatWhiteSpace ( ) : void
Finished ( ) : bool
LookAhead ( int howFar = 1 ) : char?
MakePunctuationToken ( int length ) : Token
NextChar ( ) : void
RaiseTokenizerError ( string message ) : void
Tokenize ( ) : List
TokenizeComment ( ) : void
TokenizeIdentifier ( ) : Token
TokenizeMultilineComment ( ) : void
TokenizeNumber ( ) : Token
TokenizePred ( Token type, bool>.Func pred ) : Token
TokenizePunctuation ( ) : Token
TokenizeStringLiteral ( char quote ) : Token

Method Details

Tokenizer() public method

public Tokenizer ( SourceFile source, int initialPos, int limit = -1 ) : System
source SourceFile
initialPos int
limit int
return System