• 홈
• Parrot
• Lexer
• Tokenizer

C# 클래스 Parrot.Lexer.Tokenizer

파일 보기 프로젝트 열기: ParrotFx/Parrot 1 사용 예제들

공개 메소드들

메소드	설명
Tokenizer ( Stream source ) : System
Tokens ( ) : IList

비공개 메소드들

메소드	설명
Consume ( ) : char
ConsumeIdentifier ( ) : string
ConsumeMultiCharToken ( char currentCharacter, Func contentFunc ) : Token
ConsumeQuotedStringLiteral ( char quote ) : string
ConsumeSingleCharToken ( char currentCharacter ) : Token
ConsumeToken ( TokenType type, Func contentFunc ) : Token
ConsumeUntilNewlineOrEndOfStream ( ) : string
ConsumeWhitespace ( ) : string
GetNextToken ( ) : Token
IsIdTail ( char character ) : bool
IsIdentifierHead ( char character ) : bool
IsIdentifierUnicode ( char character ) : bool
IsNewLine ( char character ) : bool
IsSingleCharToken ( char currentCharacter ) : bool
IsWhitespace ( char character ) : bool
PeekCurrentCharacter ( ) : char
Tokenize ( ) : List
initToken ( Token token, Func contentFunc ) : Token

메소드 상세