C# Class Spring.Expressions.Parser.antlr.TokenStreamRewriteEngine

This token stream tracks the *entire* token stream coming from a lexer, but does not pass on the whitespace (or whatever else you want to discard) to the parser.

This class can then be asked for the ith token in the input stream. Useful for dumping out the input stream exactly after doing some augmentation or other manipulations. Tokens are index from 0..n-1

You can insert stuff, replace, and delete chunks. Note that the operations are done lazily--only if you convert the buffer to a string. This is very efficient because you are not moving data around all the time. As the buffer of tokens is converted to strings, the toString() method(s) check to see if there is an operation at the current index. If so, the operation is done and then normal string rendering continues on the buffer. This is like having multiple Turing machine instruction streams (programs) operating on a single input tape. :)

Since the operations are done lazily at toString-time, operations do not screw up the token index values. That is, an insert operation at token index i does not change the index values for tokens i+1..n-1.

Because operations never actually alter the buffer, you may always get the original token stream back without undoing anything. Since the instructions are queued up, you can easily simulate transactions and roll back any changes if there is an error just by removing instructions. For example,

For example:

 TokenStreamRewriteEngine rewriteEngine = new TokenStreamRewriteEngine(lexer); JavaRecognizer           parser        = new JavaRecognizer(rewriteEngine); ... rewriteEngine.insertAfter("pass1", t, "foobar");} rewriteEngine.insertAfter("pass2", u, "start");} System.Console.Out.WriteLine(rewriteEngine.ToString("pass1")); System.Console.Out.WriteLine(rewriteEngine.ToString("pass2"));

You can also have multiple "instruction streams" and get multiple rewrites from a single pass over the input. Just name the instruction streams and use that name again when printing the buffer. This could be useful for generating a C file and also its header file--all from the same buffer.

If you don't use named rewrite streams, a "default" stream is used.

Terence Parr, [email protected] University of San Francisco February 2004

Inheritance: TokenStream

Show file Open project: spring-projects/spring-net

Protected Properties

Property	Type	Description
discardMask	antlr.collections.impl.BitSet
index	int
lastRewriteTokenIndexes	System.Collections.IDictionary
programs	System.Collections.IDictionary
stream	TokenStream
tokens	System.Collections.IList

Public Methods

Method	Description
ToDebugString ( ) : string
ToDebugString ( int start, int end ) : string
ToOriginalString ( ) : string
ToOriginalString ( int start, int end ) : string
ToString ( ) : string
ToString ( int start, int end ) : string
ToString ( string programName ) : string
ToString ( string programName, int start, int end ) : string
TokenStreamRewriteEngine ( TokenStream upstream )
TokenStreamRewriteEngine ( TokenStream upstream, int initialSize )
delete ( IToken indexT ) : void
delete ( IToken from, IToken to ) : void
delete ( int index ) : void
delete ( int from, int to ) : void
delete ( string programName, IToken from, IToken to ) : void
delete ( string programName, int from, int to ) : void
deleteProgram ( ) : void
deleteProgram ( string programName ) : void	Reset the program so that no instructions exist
discard ( int ttype ) : void
getLastRewriteTokenIndex ( ) : int
getToken ( int i ) : TokenWithIndex
getTokenStreamSize ( ) : int
insertAfter ( IToken t, string text ) : void
insertAfter ( int index, string text ) : void
insertAfter ( string programName, IToken t, string text ) : void
insertAfter ( string programName, int index, string text ) : void
insertBefore ( IToken t, string text ) : void
insertBefore ( int index, string text ) : void
insertBefore ( string programName, IToken t, string text ) : void
insertBefore ( string programName, int index, string text ) : void
nextToken ( ) : IToken
replace ( IToken from, IToken to, string text ) : void
replace ( IToken indexT, string text ) : void
replace ( int from, int to, string text ) : void
replace ( int index, string text ) : void
replace ( string programName, IToken from, IToken to, string text ) : void
replace ( string programName, int from, int to, string text ) : void
rollback ( int instructionIndex ) : void
rollback ( string programName, int instructionIndex ) : void	Rollback the instruction stream for a program so that the indicated instruction (via instructionIndex) is no longer in the stream. UNTESTED!

Protected Methods

Method	Description
addToSortedRewriteList ( RewriteOperation op ) : void	If op.index > lastRewriteTokenIndexes, just add to the end. Otherwise, do linear
addToSortedRewriteList ( string programName, RewriteOperation op ) : void
getLastRewriteTokenIndex ( string programName ) : int
getProgram ( string name ) : System.Collections.IList
setLastRewriteTokenIndex ( string programName, int i ) : void

Private Methods

Method	Description
initializeProgram ( string name ) : System.Collections.IList

Method Details

ToDebugString() public method

public ToDebugString ( ) : string
return	string

ToDebugString() public method

public ToDebugString ( int start, int end ) : string
start	int
end	int
return	string

ToOriginalString() public method

public ToOriginalString ( ) : string
return	string

ToOriginalString() public method

public ToOriginalString ( int start, int end ) : string
start	int
end	int
return	string

ToString() public method

public ToString ( ) : string
return	string

ToString() public method

public ToString ( int start, int end ) : string
start	int
end	int
return	string

ToString() public method

public ToString ( string programName ) : string
programName	string
return	string

ToString() public method

public ToString ( string programName, int start, int end ) : string
programName	string
start	int
end	int
return	string

TokenStreamRewriteEngine() public method

public TokenStreamRewriteEngine ( TokenStream upstream )
upstream	TokenStream

TokenStreamRewriteEngine() public method

public TokenStreamRewriteEngine ( TokenStream upstream, int initialSize )
upstream	TokenStream
initialSize	int

addToSortedRewriteList() protected method

If op.index > lastRewriteTokenIndexes, just add to the end. Otherwise, do linear

protected addToSortedRewriteList ( RewriteOperation op ) : void
op	RewriteOperation
return	void

addToSortedRewriteList() protected method

protected addToSortedRewriteList ( string programName, RewriteOperation op ) : void
programName	string
op	RewriteOperation
return	void

delete() public method

public delete ( IToken indexT ) : void
indexT	IToken
return	void

delete() public method

public delete ( IToken from, IToken to ) : void
from	IToken
to	IToken
return	void

delete() public method

public delete ( int index ) : void
index	int
return	void

delete() public method

public delete ( int from, int to ) : void
from	int
to	int
return	void

delete() public method

public delete ( string programName, IToken from, IToken to ) : void
programName	string
from	IToken
to	IToken
return	void

delete() public method

public delete ( string programName, int from, int to ) : void
programName	string
from	int
to	int
return	void

deleteProgram() public method

public deleteProgram ( ) : void
return	void

deleteProgram() public method

Reset the program so that no instructions exist

public deleteProgram ( string programName ) : void
programName	string
return	void

discard() public method

public discard ( int ttype ) : void
ttype	int
return	void

getLastRewriteTokenIndex() public method

public getLastRewriteTokenIndex ( ) : int
return	int

getLastRewriteTokenIndex() protected method

protected getLastRewriteTokenIndex ( string programName ) : int
programName	string
return	int

getProgram() protected method

protected getProgram ( string name ) : System.Collections.IList
name	string
return	System.Collections.IList

getToken() public method

public getToken ( int i ) : TokenWithIndex
i	int
return	TokenWithIndex

getTokenStreamSize() public method

public getTokenStreamSize ( ) : int
return	int

insertAfter() public method

public insertAfter ( IToken t, string text ) : void
t	IToken
text	string
return	void

insertAfter() public method

public insertAfter ( int index, string text ) : void
index	int
text	string
return	void

insertAfter() public method

public insertAfter ( string programName, IToken t, string text ) : void
programName	string
t	IToken
text	string
return	void

insertAfter() public method

public insertAfter ( string programName, int index, string text ) : void
programName	string
index	int
text	string
return	void

insertBefore() public method

public insertBefore ( IToken t, string text ) : void
t	IToken
text	string
return	void

insertBefore() public method

public insertBefore ( int index, string text ) : void
index	int
text	string
return	void

insertBefore() public method

public insertBefore ( string programName, IToken t, string text ) : void
programName	string
t	IToken
text	string
return	void

insertBefore() public method

public insertBefore ( string programName, int index, string text ) : void
programName	string
index	int
text	string
return	void

nextToken() public method

public nextToken ( ) : IToken
return	IToken

replace() public method

public replace ( IToken from, IToken to, string text ) : void
from	IToken
to	IToken
text	string
return	void

replace() public method

public replace ( IToken indexT, string text ) : void
indexT	IToken
text	string
return	void

replace() public method

public replace ( int from, int to, string text ) : void
from	int
to	int
text	string
return	void

replace() public method

public replace ( int index, string text ) : void
index	int
text	string
return	void

replace() public method

public replace ( string programName, IToken from, IToken to, string text ) : void
programName	string
from	IToken
to	IToken
text	string
return	void

replace() public method

public replace ( string programName, int from, int to, string text ) : void
programName	string
from	int
to	int
text	string
return	void

rollback() public method

public rollback ( int instructionIndex ) : void
instructionIndex	int
return	void

rollback() public method

Rollback the instruction stream for a program so that the indicated instruction (via instructionIndex) is no longer in the stream.

UNTESTED!

public rollback ( string programName, int instructionIndex ) : void
programName	string
instructionIndex	int
return	void

setLastRewriteTokenIndex() protected method

protected setLastRewriteTokenIndex ( string programName, int i ) : void
programName	string
i	int
return	void

Property Details

discardMask protected property

Which (whitespace) token(s) to throw out

protected antlr.collections.impl.BitSet discardMask
return	antlr.collections.impl.BitSet

index protected property

track index of tokens

protected int index
return	int

lastRewriteTokenIndexes protected property

Map string (program name) -> Integer index

protected System.Collections.IDictionary lastRewriteTokenIndexes
return	System.Collections.IDictionary

programs protected property

You may have multiple, named streams of rewrite operations. I'm calling these things "programs." Maps string (name) -> rewrite (List)

protected System.Collections.IDictionary programs
return	System.Collections.IDictionary

stream protected property

Who do we suck tokens from?

protected TokenStream stream
return	TokenStream

tokens protected property

Track the incoming list of tokens

protected System.Collections.IList tokens
return	System.Collections.IList