C# Class Fanx.Serial.Tokenizer

Tokenizer inputs a stream of Unicode characters and outputs tokens for the Fantom serialization grammar.

Mostra file Open project: xored/f4 Class Usage Examples

Public Properties

Property	Type	Description
m_in	Fan.Sys.InStream
m_line	int
m_type	int
m_val	object

Public Methods

Method	Description
Tokenizer ( InStream @in ) : System.Text	Construct for specified input stream.
err ( string msg ) : System.Exception
next ( ) : int	Read the next token from the stream. The token is available via the 'type' and 'val' fields. The line of the current token is available in 'line' field. Return the 'type' field or -1 if at end of stream.
reset ( int type, object val, int line ) : int	Reset the current token state.
undo ( int type, object val, int line ) : void	Pushback a token which will be the next read.

Private Methods

Method	Description
Tokenizer ( ) : System.Text
ch ( ) : int	Parse a char literal token (as Long literal).
consume ( ) : void
doNext ( ) : int	Read the next token, set the 'val' field but return type without worrying setting the 'type' field.
escape ( ) : char	Parse an escapse sequence which starts with a \
hex ( ) : int	Process hex int/long literal starting with 0x
hex ( int c ) : int
id ( ) : int	Parse an identifier: alpha (alpha\|number)*
number ( bool neg ) : int	Parse a number literal token.
skipCommentML ( ) : Token	Skip a multi line /* comment. Note unlike C/Java, slash/star comments can be nested.
skipCommentSL ( ) : Token	Skip a single line // comment
str ( ) : int	Parse a string literal token.
uri ( ) : int	Parse a uri literal token.

Method Details

Tokenizer() public method

Construct for specified input stream.

public Tokenizer ( InStream @in ) : System.Text
@in	Fan.Sys.InStream
return	System.Text

err() public method

public err ( string msg ) : System.Exception
msg	string
return	System.Exception

next() public method

Read the next token from the stream. The token is available via the 'type' and 'val' fields. The line of the current token is available in 'line' field. Return the 'type' field or -1 if at end of stream.

public next ( ) : int
return	int

reset() public method

Reset the current token state.

public reset ( int type, object val, int line ) : int
type	int
val	object
line	int
return	int

undo() public method

Pushback a token which will be the next read.

public undo ( int type, object val, int line ) : void
type	int
val	object
line	int
return	void

Property Details

m_in public_oe property

public InStream,Fan.Sys m_in
return	Fan.Sys.InStream

m_line public_oe property

public int m_line
return	int

m_type public_oe property

public int m_type
return	int

m_val public_oe property

public object m_val
return	object