C# Class Spring.Expressions.Parser.antlr.TokenStreamRewriteEngine

This token stream tracks the *entire* token stream coming from a lexer, but does not pass on the whitespace (or whatever else you want to discard) to the parser.

This class can then be asked for the ith token in the input stream. Useful for dumping out the input stream exactly after doing some augmentation or other manipulations. Tokens are index from 0..n-1

You can insert stuff, replace, and delete chunks. Note that the operations are done lazily--only if you convert the buffer to a string. This is very efficient because you are not moving data around all the time. As the buffer of tokens is converted to strings, the toString() method(s) check to see if there is an operation at the current index. If so, the operation is done and then normal string rendering continues on the buffer. This is like having multiple Turing machine instruction streams (programs) operating on a single input tape. :)

Since the operations are done lazily at toString-time, operations do not screw up the token index values. That is, an insert operation at token index i does not change the index values for tokens i+1..n-1.

Because operations never actually alter the buffer, you may always get the original token stream back without undoing anything. Since the instructions are queued up, you can easily simulate transactions and roll back any changes if there is an error just by removing instructions. For example,

For example: TokenStreamRewriteEngine rewriteEngine = new TokenStreamRewriteEngine(lexer); JavaRecognizer parser = new JavaRecognizer(rewriteEngine); ... rewriteEngine.insertAfter("pass1", t, "foobar");} rewriteEngine.insertAfter("pass2", u, "start");} System.Console.Out.WriteLine(rewriteEngine.ToString("pass1")); System.Console.Out.WriteLine(rewriteEngine.ToString("pass2"));

You can also have multiple "instruction streams" and get multiple rewrites from a single pass over the input. Just name the instruction streams and use that name again when printing the buffer. This could be useful for generating a C file and also its header file--all from the same buffer.

If you don't use named rewrite streams, a "default" stream is used.

Terence Parr, [email protected] University of San Francisco February 2004

Inheritance: TokenStream
Datei anzeigen Open project: spring-projects/spring-net

Protected Properties

Property Type Description
discardMask antlr.collections.impl.BitSet
index int
lastRewriteTokenIndexes System.Collections.IDictionary
programs System.Collections.IDictionary
stream TokenStream
tokens System.Collections.IList

Public Methods

Method Description
ToDebugString ( ) : string
ToDebugString ( int start, int end ) : string
ToOriginalString ( ) : string
ToOriginalString ( int start, int end ) : string
ToString ( ) : string
ToString ( int start, int end ) : string
ToString ( string programName ) : string
ToString ( string programName, int start, int end ) : string
TokenStreamRewriteEngine ( TokenStream upstream )
TokenStreamRewriteEngine ( TokenStream upstream, int initialSize )
delete ( IToken indexT ) : void
delete ( IToken from, IToken to ) : void
delete ( int index ) : void
delete ( int from, int to ) : void
delete ( string programName, IToken from, IToken to ) : void
delete ( string programName, int from, int to ) : void
deleteProgram ( ) : void
deleteProgram ( string programName ) : void

Reset the program so that no instructions exist
discard ( int ttype ) : void
getLastRewriteTokenIndex ( ) : int
getToken ( int i ) : TokenWithIndex
getTokenStreamSize ( ) : int
insertAfter ( IToken t, string text ) : void
insertAfter ( int index, string text ) : void
insertAfter ( string programName, IToken t, string text ) : void
insertAfter ( string programName, int index, string text ) : void
insertBefore ( IToken t, string text ) : void
insertBefore ( int index, string text ) : void
insertBefore ( string programName, IToken t, string text ) : void
insertBefore ( string programName, int index, string text ) : void
nextToken ( ) : IToken
replace ( IToken from, IToken to, string text ) : void
replace ( IToken indexT, string text ) : void
replace ( int from, int to, string text ) : void
replace ( int index, string text ) : void
replace ( string programName, IToken from, IToken to, string text ) : void
replace ( string programName, int from, int to, string text ) : void
rollback ( int instructionIndex ) : void
rollback ( string programName, int instructionIndex ) : void

Rollback the instruction stream for a program so that the indicated instruction (via instructionIndex) is no longer in the stream.

UNTESTED!

Protected Methods

Method Description
addToSortedRewriteList ( RewriteOperation op ) : void

If op.index > lastRewriteTokenIndexes, just add to the end. Otherwise, do linear
addToSortedRewriteList ( string programName, RewriteOperation op ) : void
getLastRewriteTokenIndex ( string programName ) : int
getProgram ( string name ) : System.Collections.IList
setLastRewriteTokenIndex ( string programName, int i ) : void

Private Methods

Method Description
initializeProgram ( string name ) : System.Collections.IList

Method Details

ToDebugString() public method

public ToDebugString ( ) : string
return string

ToDebugString() public method

public ToDebugString ( int start, int end ) : string
start int
end int
return string

ToOriginalString() public method

public ToOriginalString ( ) : string
return string

ToOriginalString() public method

public ToOriginalString ( int start, int end ) : string
start int
end int
return string

ToString() public method

public ToString ( ) : string
return string

ToString() public method

public ToString ( int start, int end ) : string
start int
end int
return string

ToString() public method

public ToString ( string programName ) : string
programName string
return string

ToString() public method

public ToString ( string programName, int start, int end ) : string
programName string
start int
end int
return string

TokenStreamRewriteEngine() public method

public TokenStreamRewriteEngine ( TokenStream upstream )
upstream TokenStream

TokenStreamRewriteEngine() public method

public TokenStreamRewriteEngine ( TokenStream upstream, int initialSize )
upstream TokenStream
initialSize int

addToSortedRewriteList() protected method

If op.index > lastRewriteTokenIndexes, just add to the end. Otherwise, do linear
protected addToSortedRewriteList ( RewriteOperation op ) : void
op RewriteOperation
return void

addToSortedRewriteList() protected method

protected addToSortedRewriteList ( string programName, RewriteOperation op ) : void
programName string
op RewriteOperation
return void

delete() public method

public delete ( IToken indexT ) : void
indexT IToken
return void

delete() public method

public delete ( IToken from, IToken to ) : void
from IToken
to IToken
return void

delete() public method

public delete ( int index ) : void
index int
return void

delete() public method

public delete ( int from, int to ) : void
from int
to int
return void

delete() public method

public delete ( string programName, IToken from, IToken to ) : void
programName string
from IToken
to IToken
return void

delete() public method

public delete ( string programName, int from, int to ) : void
programName string
from int
to int
return void

deleteProgram() public method

public deleteProgram ( ) : void
return void

deleteProgram() public method

Reset the program so that no instructions exist
public deleteProgram ( string programName ) : void
programName string
return void

discard() public method

public discard ( int ttype ) : void
ttype int
return void

getLastRewriteTokenIndex() public method

public getLastRewriteTokenIndex ( ) : int
return int

getLastRewriteTokenIndex() protected method

protected getLastRewriteTokenIndex ( string programName ) : int
programName string
return int

getProgram() protected method

protected getProgram ( string name ) : System.Collections.IList
name string
return System.Collections.IList

getToken() public method

public getToken ( int i ) : TokenWithIndex
i int
return TokenWithIndex

getTokenStreamSize() public method

public getTokenStreamSize ( ) : int
return int

insertAfter() public method

public insertAfter ( IToken t, string text ) : void
t IToken
text string
return void

insertAfter() public method

public insertAfter ( int index, string text ) : void
index int
text string
return void

insertAfter() public method

public insertAfter ( string programName, IToken t, string text ) : void
programName string
t IToken
text string
return void

insertAfter() public method

public insertAfter ( string programName, int index, string text ) : void
programName string
index int
text string
return void

insertBefore() public method

public insertBefore ( IToken t, string text ) : void
t IToken
text string
return void

insertBefore() public method

public insertBefore ( int index, string text ) : void
index int
text string
return void

insertBefore() public method

public insertBefore ( string programName, IToken t, string text ) : void
programName string
t IToken
text string
return void

insertBefore() public method

public insertBefore ( string programName, int index, string text ) : void
programName string
index int
text string
return void

nextToken() public method

public nextToken ( ) : IToken
return IToken

replace() public method

public replace ( IToken from, IToken to, string text ) : void
from IToken
to IToken
text string
return void

replace() public method

public replace ( IToken indexT, string text ) : void
indexT IToken
text string
return void

replace() public method

public replace ( int from, int to, string text ) : void
from int
to int
text string
return void

replace() public method

public replace ( int index, string text ) : void
index int
text string
return void

replace() public method

public replace ( string programName, IToken from, IToken to, string text ) : void
programName string
from IToken
to IToken
text string
return void

replace() public method

public replace ( string programName, int from, int to, string text ) : void
programName string
from int
to int
text string
return void

rollback() public method

public rollback ( int instructionIndex ) : void
instructionIndex int
return void

rollback() public method

Rollback the instruction stream for a program so that the indicated instruction (via instructionIndex) is no longer in the stream.
UNTESTED!
public rollback ( string programName, int instructionIndex ) : void
programName string
instructionIndex int
return void

setLastRewriteTokenIndex() protected method

protected setLastRewriteTokenIndex ( string programName, int i ) : void
programName string
i int
return void

Property Details

discardMask protected_oe property

Which (whitespace) token(s) to throw out
protected antlr.collections.impl.BitSet discardMask
return antlr.collections.impl.BitSet

index protected_oe property

track index of tokens
protected int index
return int

lastRewriteTokenIndexes protected_oe property

Map string (program name) -> Integer index
protected System.Collections.IDictionary lastRewriteTokenIndexes
return System.Collections.IDictionary

programs protected_oe property

You may have multiple, named streams of rewrite operations. I'm calling these things "programs." Maps string (name) -> rewrite (List)
protected System.Collections.IDictionary programs
return System.Collections.IDictionary

stream protected_oe property

Who do we suck tokens from?
protected TokenStream stream
return TokenStream

tokens protected_oe property

Track the incoming list of tokens
protected System.Collections.IList tokens
return System.Collections.IList