• Home
• Parrot
• Lexer
• Tokenizer

C# Class Parrot.Lexer.Tokenizer

Show file Open project: ParrotFx/Parrot Class Usage Examples

Public Methods

Method	Description
Tokenizer ( Stream source ) : System
Tokens ( ) : IList

Private Methods

Method	Description
Consume ( ) : char
ConsumeIdentifier ( ) : string
ConsumeMultiCharToken ( char currentCharacter, Func contentFunc ) : Token
ConsumeQuotedStringLiteral ( char quote ) : string
ConsumeSingleCharToken ( char currentCharacter ) : Token
ConsumeToken ( TokenType type, Func contentFunc ) : Token
ConsumeUntilNewlineOrEndOfStream ( ) : string
ConsumeWhitespace ( ) : string
GetNextToken ( ) : Token
IsIdTail ( char character ) : bool
IsIdentifierHead ( char character ) : bool
IsIdentifierUnicode ( char character ) : bool
IsNewLine ( char character ) : bool
IsSingleCharToken ( char currentCharacter ) : bool
IsWhitespace ( char character ) : bool
PeekCurrentCharacter ( ) : char
Tokenize ( ) : List
initToken ( Token token, Func contentFunc ) : Token

Method Details