C# Class Antlr4.Runtime.BufferedTokenStream

This implementation of ITokenStream loads tokens from a ITokenSource on-demand, and places the tokens in a buffer to provide access to any previous token by index.

This token stream ignores the value of IToken.Channel() . If your parser requires the token stream filter tokens to only those on a particular channel, such as TokenConstants.DefaultChannel or TokenConstants.HiddenChannel , use a filtering token stream such a CommonTokenStream .

Inheritance: ITokenStream

Mostrar archivo Open project: antlr/antlr4 Class Usage Examples

Protected Properties

Property	Type	Description
fetchedEOF	bool
p	int
tokens	IList

Public Methods

Method	Description
BufferedTokenStream ( ITokenSource tokenSource ) : System
Consume ( ) : void
Fill ( ) : void	Get all tokens from lexer until EOF. Get all tokens from lexer until EOF.
Get ( int start, int stop ) : IList	Get all tokens from start..stop inclusively. Get all tokens from start..stop inclusively.
Get ( int i ) : IToken
GetHiddenTokensToLeft ( int tokenIndex ) : IList	Collect all hidden tokens (any off-default channel) to the left of the current token up until we see a token on Lexer.DefaultTokenChannel .
GetHiddenTokensToLeft ( int tokenIndex, int channel ) : IList	Collect all tokens on specified channel to the left of the current token up until we see a token on Lexer.DefaultTokenChannel . If channel is `-1` , find any non default channel token.
GetHiddenTokensToRight ( int tokenIndex ) : IList	Collect all hidden tokens (any off-default channel) to the right of the current token up until we see a token on Lexer.DefaultTokenChannel or EOF.
GetHiddenTokensToRight ( int tokenIndex, int channel ) : IList	Collect all tokens on specified channel to the right of the current token up until we see a token on Lexer.DefaultTokenChannel or EOF. If channel is `-1` , find any non default channel token.
GetTokens ( ) : IList
GetTokens ( int start, int stop ) : IList
GetTokens ( int start, int stop, BitSet types ) : IList	Given a start and stop index, return a `List` of all tokens in the token type `BitSet` . Return if no tokens were found. This method looks at both on and off channel tokens.
GetTokens ( int start, int stop, int ttype ) : IList
LA ( int i ) : int
Mark ( ) : int
Release ( int marker ) : void
Reset ( ) : void
Seek ( int index ) : void
SetTokenSource ( ITokenSource tokenSource ) : void	Reset this token stream by setting its token source. Reset this token stream by setting its token source.

Protected Methods

Method	Description
AdjustSeekIndex ( int i ) : int	Allowed derived classes to modify the behavior of operations which change the current stream position by adjusting the target token index of a seek operation. Allowed derived classes to modify the behavior of operations which change the current stream position by adjusting the target token index of a seek operation. The default implementation simply returns i . If an exception is thrown in this method, the current stream index should not be changed. For example, CommonTokenStream overrides this method to ensure that the seek target is always an on-channel token.
Fetch ( int n ) : int	Add n elements to buffer.
FilterForChannel ( int from, int to, int channel ) : IList
LazyInit ( ) : void
Lb ( int k ) : IToken
NextTokenOnChannel ( int i, int channel ) : int	Given a starting index, return the index of the next token on channel. Given a starting index, return the index of the next token on channel. Return i if `tokens[i]` is on channel. Return the index of the EOF token if there are no tokens on channel between i and EOF.
PreviousTokenOnChannel ( int i, int channel ) : int	Given a starting index, return the index of the previous token on channel. Given a starting index, return the index of the previous token on channel. Return i if `tokens[i]` is on channel. Return -1 if there are no tokens on channel between i and 0. If i specifies an index at or after the EOF token, the EOF token index is returned. This is due to the fact that the EOF token is treated as though it were on every channel.
Setup ( ) : void
Sync ( int i ) : bool	Make sure index i in tokens has a token.

Private Methods

Method	Description
GetText ( ) : string
GetText ( IToken start, IToken stop ) : string
GetText ( Interval interval ) : string
GetText ( RuleContext ctx ) : string
LT ( int k ) : IToken

Method Details

AdjustSeekIndex() protected method

Allowed derived classes to modify the behavior of operations which change the current stream position by adjusting the target token index of a seek operation.

Allowed derived classes to modify the behavior of operations which change the current stream position by adjusting the target token index of a seek operation. The default implementation simply returns i . If an exception is thrown in this method, the current stream index should not be changed.

For example, CommonTokenStream overrides this method to ensure that the seek target is always an on-channel token.

protected AdjustSeekIndex ( int i ) : int
i	int	The target token index.
return	int

BufferedTokenStream() public method

public BufferedTokenStream ( ITokenSource tokenSource ) : System
tokenSource	ITokenSource
return	System

Consume() public method

public Consume ( ) : void
return	void

Fetch() protected method

Add n elements to buffer.

protected Fetch ( int n ) : int
n	int
return	int

Fill() public method

Get all tokens from lexer until EOF.

public Fill ( ) : void
return	void

FilterForChannel() protected method

protected FilterForChannel ( int from, int to, int channel ) : IList
from	int
to	int
channel	int
return	IList

Get() public method

Get all tokens from start..stop inclusively.

public Get ( int start, int stop ) : IList
start	int
stop	int
return	IList

Get() public method

public Get ( int i ) : IToken
i	int
return	IToken

GetHiddenTokensToLeft() public method

Collect all hidden tokens (any off-default channel) to the left of the current token up until we see a token on Lexer.DefaultTokenChannel .

public GetHiddenTokensToLeft ( int tokenIndex ) : IList
tokenIndex	int
return	IList

GetHiddenTokensToLeft() public method

Collect all tokens on specified channel to the left of the current token up until we see a token on Lexer.DefaultTokenChannel . If channel is -1 , find any non default channel token.

public GetHiddenTokensToLeft ( int tokenIndex, int channel ) : IList
tokenIndex	int
channel	int
return	IList

GetHiddenTokensToRight() public method

Collect all hidden tokens (any off-default channel) to the right of the current token up until we see a token on Lexer.DefaultTokenChannel or EOF.

public GetHiddenTokensToRight ( int tokenIndex ) : IList
tokenIndex	int
return	IList

GetHiddenTokensToRight() public method

Collect all tokens on specified channel to the right of the current token up until we see a token on Lexer.DefaultTokenChannel or EOF. If channel is -1 , find any non default channel token.

public GetHiddenTokensToRight ( int tokenIndex, int channel ) : IList
tokenIndex	int
channel	int
return	IList

GetTokens() public method

public GetTokens ( ) : IList
return	IList

GetTokens() public method

public GetTokens ( int start, int stop ) : IList
start	int
stop	int
return	IList

GetTokens() public method

Given a start and stop index, return a List of all tokens in the token type BitSet . Return if no tokens were found. This method looks at both on and off channel tokens.

public GetTokens ( int start, int stop, BitSet types ) : IList
start	int
stop	int
types	Antlr4.Runtime.Sharpen.BitSet
return	IList

GetTokens() public method

public GetTokens ( int start, int stop, int ttype ) : IList
start	int
stop	int
ttype	int
return	IList

LA() public method

public LA ( int i ) : int
i	int
return	int

LazyInit() protected method

protected LazyInit ( ) : void
return	void

Lb() protected method

protected Lb ( int k ) : IToken
k	int
return	IToken

Mark() public method

public Mark ( ) : int
return	int

NextTokenOnChannel() protected method

Given a starting index, return the index of the next token on channel.

Given a starting index, return the index of the next token on channel. Return i if tokens[i] is on channel. Return the index of the EOF token if there are no tokens on channel between i and EOF.

protected NextTokenOnChannel ( int i, int channel ) : int
i	int
channel	int
return	int

PreviousTokenOnChannel() protected method

Given a starting index, return the index of the previous token on channel.

Given a starting index, return the index of the previous token on channel. Return i if tokens[i] is on channel. Return -1 if there are no tokens on channel between i and 0.

If i specifies an index at or after the EOF token, the EOF token index is returned. This is due to the fact that the EOF token is treated as though it were on every channel.

protected PreviousTokenOnChannel ( int i, int channel ) : int
i	int
channel	int
return	int

Release() public method

public Release ( int marker ) : void
marker	int
return	void

Reset() public method

public Reset ( ) : void
return	void

Seek() public method

public Seek ( int index ) : void
index	int
return	void

SetTokenSource() public method

Reset this token stream by setting its token source.

public SetTokenSource ( ITokenSource tokenSource ) : void
tokenSource	ITokenSource
return	void

Setup() protected method

protected Setup ( ) : void
return	void

Sync() protected method

Make sure index i in tokens has a token.

protected Sync ( int i ) : bool
i	int
return	bool

Property Details

fetchedEOF protected_oe property

Indicates whether the TokenConstants.EOF token has been fetched from _tokenSource and added to tokens . This field improves performance for the following cases:

Consume() : The lookahead check in Consume() to prevent consuming the EOF symbol is optimized by checking the values of fetchedEOF and p instead of calling LA(int) .
Fetch(int) : The check to prevent adding multiple EOF symbols into tokens is trivial with this field.

protected bool fetchedEOF
return	bool

p protected_oe property

The index into tokens of the current token (next token to Consume() ). tokens [ p ] should be LT(1) .

This field is set to -1 when the stream is first constructed or when SetTokenSource(ITokenSource) is called, indicating that the first token has not yet been fetched from the token source. For additional information, see the documentation of IIntStream for a description of Initializing Methods.

protected int p
return	int

tokens protected_oe property

A collection of all tokens fetched from the token source.

A collection of all tokens fetched from the token source. The list is considered a complete view of the input once fetchedEOF is set to .

protected IList tokens
return	IList