C# Class Microsoft.Scripting.Hosting.TokenCategorizer

Inheritance: System.Object

Public Methods

Method	Description
Initialize ( object state, ScriptSource scriptSource, SourceLocation initialLocation ) : void
InitializeLifetimeService ( ) : object
ReadToken ( ) : TokenInfo	Move the tokenizer past the next token and return its category.
ReadTokens ( int characterCount ) : IEnumerable	Get all tokens over a block of the stream. The scanner should return full tokens. If startLocation + length lands in the middle of a token, the full token should be returned.
SkipToken ( ) : bool	Move the tokenizer past the next token.
SkipTokens ( int characterCount ) : bool	Scan from startLocation to at least startLocation + length. This method is used to determine state at arbitrary startLocation.

Method	Description
TokenCategorizer ( Microsoft.Scripting.Runtime.TokenizerService tokenizer ) : System

public Initialize ( object state, ScriptSource scriptSource, SourceLocation initialLocation ) : void
state	object
scriptSource	ScriptSource
initialLocation	SourceLocation
return	void

public InitializeLifetimeService ( ) : object
return	object

Move the tokenizer past the next token and return its category.

public ReadToken ( ) : TokenInfo
return	TokenInfo

Get all tokens over a block of the stream.

The scanner should return full tokens. If startLocation + length lands in the middle of a token, the full token should be returned.

public ReadTokens ( int characterCount ) : IEnumerable
characterCount	int	Tokens are read until at least given amount of characters is read or the stream ends.
return	IEnumerable

Move the tokenizer past the next token.

public SkipToken ( ) : bool
return	bool

Scan from startLocation to at least startLocation + length.

This method is used to determine state at arbitrary startLocation.

public SkipTokens ( int characterCount ) : bool
characterCount	int	Tokens are read until at least given amount of characters is read or the stream ends.
return	bool