C# Class _3PA.MainFeatures.Parser.Lexer

This class "tokenize" the input data into tokens of various types, it implements a visitor pattern

Show file Open project: jcaillon/3P Class Usage Examples

Public Methods

Method	Description
Accept ( ILexerVisitor visitor ) : void	Feed this method with a visitor implementing ILexerVisitor to visit all the tokens of the input string (you must call the Tokenize() methode before that!)
Lexer ( string data ) : System	constructor, data is the input string to tokenize call Tokenize() to do the work
Lexer ( string data, int pos, int line, int column, int commentDepth, int includeDepth ) : System	Use this when you wish to tokenize only a partial string in a longer string Allows you to start with a comment depth different of 0
MoveNextToken ( ) : bool	To use this lexer as an enumerator, Move to the next token, return true if it can
PeekAtToken ( int x ) : Token	To use this lexer as an enumerator, peek at the current pos + x token of the list, returns a new TokenEof if can't find

Private Methods

Method	Description
CreateCommentToken ( ) : Token	create a comment token, take into account nested comment, reads until all comments tags are closed
CreateEolToken ( char ch ) : Token
CreateEosToken ( ) : Token
CreateIncludeToken ( ) : Token	reads an include declaration
CreateNumberToken ( ) : Token	create a number token, accepts decimal value with a '.' and hexadecimal notation 0xNNN
CreatePreProcessedStatement ( ) : Token	reads a preproc definition
CreateSingleLineCommentToken ( ) : Token	create a comment token, for a single line comment
CreateStringDescriptorToken ( ) : Token	A character-string in progress can be described with different properties : "characters" [ : [ R \| L \| C \| T ] [ U ] [ max-length ] ]
CreateStringToken ( char strChar ) : Token	reads a quotes string (either simple of double quote), takes into account escape char ~
CreateSymbolToken ( ) : Token
CreateWhitespaceToken ( ) : Token	create a whitespace token (successions of either ' ' or '\t')
GetNext ( ) : Token	returns the next token of the string
GetTokenValue ( ) : string	Returns the current value of the token
PeekAt ( int x ) : char	Peek forward x chars
PeekAtReverse ( int x ) : char	peek backward x chars
Read ( ) : void	Read to the next char, indirectly adding the current char (_data[_pos]) to the current token
ReadEol ( char eol = '\n' ) : void	Read the End Of Line character (can read \r\n in one go), add it to the current token
ReadWord ( ) : void	reads a word with this format : [a-Z_&]+[\w_-]((\.[\w_-])?){1,}
Tokenize ( ) : void	Call this method to actually tokenize the string

Method Details

Accept() public method

Feed this method with a visitor implementing ILexerVisitor to visit all the tokens of the input string (you must call the Tokenize() methode before that!)

public Accept ( ILexerVisitor visitor ) : void
visitor	ILexerVisitor
return	void

Lexer() public method

constructor, data is the input string to tokenize call Tokenize() to do the work

public Lexer ( string data ) : System
data	string
return	System

Lexer() public method

Use this when you wish to tokenize only a partial string in a longer string Allows you to start with a comment depth different of 0

public Lexer ( string data, int pos, int line, int column, int commentDepth, int includeDepth ) : System
data	string
pos	int
line	int
column	int
commentDepth	int
includeDepth	int
return	System

MoveNextToken() public method

To use this lexer as an enumerator, Move to the next token, return true if it can

public MoveNextToken ( ) : bool
return	bool

PeekAtToken() public method

To use this lexer as an enumerator, peek at the current pos + x token of the list, returns a new TokenEof if can't find

public PeekAtToken ( int x ) : Token
x	int
return	Token