C# Class Antlr4.Runtime.BufferedTokenStream

This implementation of ITokenStream loads tokens from a ITokenSource on-demand, and places the tokens in a buffer to provide access to any previous token by index.

This token stream ignores the value of IToken.Channel() . If your parser requires the token stream filter tokens to only those on a particular channel, such as TokenConstants.DefaultChannel or TokenConstants.HiddenChannel , use a filtering token stream such a CommonTokenStream .

Inheritance: ITokenStream
Show file Open project: antlr/antlr4 Class Usage Examples

Protected Properties

Property Type Description
fetchedEOF bool
p int
tokens IList

Public Methods

Method Description
BufferedTokenStream ( ITokenSource tokenSource ) : System
Consume ( ) : void
Fill ( ) : void

Get all tokens from lexer until EOF.

Get all tokens from lexer until EOF.

Get ( int start, int stop ) : IList

Get all tokens from start..stop inclusively.

Get all tokens from start..stop inclusively.

Get ( int i ) : IToken
GetHiddenTokensToLeft ( int tokenIndex ) : IList

Collect all hidden tokens (any off-default channel) to the left of the current token up until we see a token on Lexer.DefaultTokenChannel .

GetHiddenTokensToLeft ( int tokenIndex, int channel ) : IList

Collect all tokens on specified channel to the left of the current token up until we see a token on Lexer.DefaultTokenChannel . If channel is -1 , find any non default channel token.

GetHiddenTokensToRight ( int tokenIndex ) : IList

Collect all hidden tokens (any off-default channel) to the right of the current token up until we see a token on Lexer.DefaultTokenChannel or EOF.

GetHiddenTokensToRight ( int tokenIndex, int channel ) : IList

Collect all tokens on specified channel to the right of the current token up until we see a token on Lexer.DefaultTokenChannel or EOF. If channel is -1 , find any non default channel token.

GetTokens ( ) : IList
GetTokens ( int start, int stop ) : IList
GetTokens ( int start, int stop, BitSet types ) : IList

Given a start and stop index, return a List of all tokens in the token type BitSet . Return if no tokens were found. This method looks at both on and off channel tokens.

GetTokens ( int start, int stop, int ttype ) : IList
LA ( int i ) : int
Mark ( ) : int
Release ( int marker ) : void
Reset ( ) : void
Seek ( int index ) : void
SetTokenSource ( ITokenSource tokenSource ) : void

Reset this token stream by setting its token source.

Reset this token stream by setting its token source.

Protected Methods

Method Description
AdjustSeekIndex ( int i ) : int

Allowed derived classes to modify the behavior of operations which change the current stream position by adjusting the target token index of a seek operation.

Allowed derived classes to modify the behavior of operations which change the current stream position by adjusting the target token index of a seek operation. The default implementation simply returns i . If an exception is thrown in this method, the current stream index should not be changed.

For example, CommonTokenStream overrides this method to ensure that the seek target is always an on-channel token.

Fetch ( int n ) : int

Add n elements to buffer.

FilterForChannel ( int from, int to, int channel ) : IList
LazyInit ( ) : void
Lb ( int k ) : IToken
NextTokenOnChannel ( int i, int channel ) : int

Given a starting index, return the index of the next token on channel.

Given a starting index, return the index of the next token on channel. Return i if tokens[i] is on channel. Return the index of the EOF token if there are no tokens on channel between i and EOF.

PreviousTokenOnChannel ( int i, int channel ) : int

Given a starting index, return the index of the previous token on channel.

Given a starting index, return the index of the previous token on channel. Return i if tokens[i] is on channel. Return -1 if there are no tokens on channel between i and 0.

If i specifies an index at or after the EOF token, the EOF token index is returned. This is due to the fact that the EOF token is treated as though it were on every channel.

Setup ( ) : void
Sync ( int i ) : bool

Make sure index i in tokens has a token.

Private Methods

Method Description
GetText ( ) : string
GetText ( IToken start, IToken stop ) : string
GetText ( Interval interval ) : string
GetText ( RuleContext ctx ) : string
LT ( int k ) : IToken

Method Details

AdjustSeekIndex() protected method

Allowed derived classes to modify the behavior of operations which change the current stream position by adjusting the target token index of a seek operation.
Allowed derived classes to modify the behavior of operations which change the current stream position by adjusting the target token index of a seek operation. The default implementation simply returns i . If an exception is thrown in this method, the current stream index should not be changed.

For example, CommonTokenStream overrides this method to ensure that the seek target is always an on-channel token.

protected AdjustSeekIndex ( int i ) : int
i int The target token index.
return int

BufferedTokenStream() public method

public BufferedTokenStream ( ITokenSource tokenSource ) : System
tokenSource ITokenSource
return System

Consume() public method

public Consume ( ) : void
return void

Fetch() protected method

Add n elements to buffer.
protected Fetch ( int n ) : int
n int
return int

Fill() public method

Get all tokens from lexer until EOF.
Get all tokens from lexer until EOF.
public Fill ( ) : void
return void

FilterForChannel() protected method

protected FilterForChannel ( int from, int to, int channel ) : IList
from int
to int
channel int
return IList

Get() public method

Get all tokens from start..stop inclusively.
Get all tokens from start..stop inclusively.
public Get ( int start, int stop ) : IList
start int
stop int
return IList

Get() public method

public Get ( int i ) : IToken
i int
return IToken

GetHiddenTokensToLeft() public method

Collect all hidden tokens (any off-default channel) to the left of the current token up until we see a token on Lexer.DefaultTokenChannel .
public GetHiddenTokensToLeft ( int tokenIndex ) : IList
tokenIndex int
return IList

GetHiddenTokensToLeft() public method

Collect all tokens on specified channel to the left of the current token up until we see a token on Lexer.DefaultTokenChannel . If channel is -1 , find any non default channel token.
public GetHiddenTokensToLeft ( int tokenIndex, int channel ) : IList
tokenIndex int
channel int
return IList

GetHiddenTokensToRight() public method

Collect all hidden tokens (any off-default channel) to the right of the current token up until we see a token on Lexer.DefaultTokenChannel or EOF.
public GetHiddenTokensToRight ( int tokenIndex ) : IList
tokenIndex int
return IList

GetHiddenTokensToRight() public method

Collect all tokens on specified channel to the right of the current token up until we see a token on Lexer.DefaultTokenChannel or EOF. If channel is -1 , find any non default channel token.
public GetHiddenTokensToRight ( int tokenIndex, int channel ) : IList
tokenIndex int
channel int
return IList

GetTokens() public method

public GetTokens ( ) : IList
return IList

GetTokens() public method

public GetTokens ( int start, int stop ) : IList
start int
stop int
return IList

GetTokens() public method

Given a start and stop index, return a List of all tokens in the token type BitSet . Return if no tokens were found. This method looks at both on and off channel tokens.
public GetTokens ( int start, int stop, BitSet types ) : IList
start int
stop int
types Antlr4.Runtime.Sharpen.BitSet
return IList

GetTokens() public method

public GetTokens ( int start, int stop, int ttype ) : IList
start int
stop int
ttype int
return IList

LA() public method

public LA ( int i ) : int
i int
return int

LazyInit() protected method

protected LazyInit ( ) : void
return void

Lb() protected method

protected Lb ( int k ) : IToken
k int
return IToken

Mark() public method

public Mark ( ) : int
return int

NextTokenOnChannel() protected method

Given a starting index, return the index of the next token on channel.
Given a starting index, return the index of the next token on channel. Return i if tokens[i] is on channel. Return the index of the EOF token if there are no tokens on channel between i and EOF.
protected NextTokenOnChannel ( int i, int channel ) : int
i int
channel int
return int

PreviousTokenOnChannel() protected method

Given a starting index, return the index of the previous token on channel.
Given a starting index, return the index of the previous token on channel. Return i if tokens[i] is on channel. Return -1 if there are no tokens on channel between i and 0.

If i specifies an index at or after the EOF token, the EOF token index is returned. This is due to the fact that the EOF token is treated as though it were on every channel.

protected PreviousTokenOnChannel ( int i, int channel ) : int
i int
channel int
return int

Release() public method

public Release ( int marker ) : void
marker int
return void

Reset() public method

public Reset ( ) : void
return void

Seek() public method

public Seek ( int index ) : void
index int
return void

SetTokenSource() public method

Reset this token stream by setting its token source.
Reset this token stream by setting its token source.
public SetTokenSource ( ITokenSource tokenSource ) : void
tokenSource ITokenSource
return void

Setup() protected method

protected Setup ( ) : void
return void

Sync() protected method

Make sure index i in tokens has a token.
protected Sync ( int i ) : bool
i int
return bool

Property Details

fetchedEOF protected property

Indicates whether the TokenConstants.EOF token has been fetched from _tokenSource and added to tokens . This field improves performance for the following cases:
  • Consume() : The lookahead check in Consume() to prevent consuming the EOF symbol is optimized by checking the values of fetchedEOF and p instead of calling LA(int) .
  • Fetch(int) : The check to prevent adding multiple EOF symbols into tokens is trivial with this field.
protected bool fetchedEOF
return bool

p protected property

The index into tokens of the current token (next token to Consume() ). tokens [ p ] should be LT(1) .

This field is set to -1 when the stream is first constructed or when SetTokenSource(ITokenSource) is called, indicating that the first token has not yet been fetched from the token source. For additional information, see the documentation of IIntStream for a description of Initializing Methods.

protected int p
return int

tokens protected property

A collection of all tokens fetched from the token source.
A collection of all tokens fetched from the token source. The list is considered a complete view of the input once fetchedEOF is set to .
protected IList tokens
return IList