Inheritance diagram for antlr.TokenStreamRewriteEngine:

Classes
class	DeleteOp
class	InsertBeforeOp
class	ReplaceOp
class	RewriteOperation
Public Member Functions
	TokenStreamRewriteEngine (TokenStream upstream)
	TokenStreamRewriteEngine (TokenStream upstream, int initialSize)
Token	nextToken () throws TokenStreamException
void	rollback (int instructionIndex)
void	rollback (String programName, int instructionIndex)
void	deleteProgram ()
void	deleteProgram (String programName)
void	insertAfter (Token t, String text)
void	insertAfter (int index, String text)
void	insertAfter (String programName, Token t, String text)
void	insertAfter (String programName, int index, String text)
void	insertBefore (Token t, String text)
void	insertBefore (int index, String text)
void	insertBefore (String programName, Token t, String text)
void	insertBefore (String programName, int index, String text)
void	replace (int index, String text)
void	replace (int from, int to, String text)
void	replace (Token indexT, String text)
void	replace (Token from, Token to, String text)
void	replace (String programName, int from, int to, String text)
void	replace (String programName, Token from, Token to, String text)
void	delete (int index)
void	delete (int from, int to)
void	delete (Token indexT)
void	delete (Token from, Token to)
void	delete (String programName, int from, int to)
void	delete (String programName, Token from, Token to)
void	discard (int ttype)
TokenWithIndex	getToken (int i)
int	getTokenStreamSize ()
String	toOriginalString ()
String	toOriginalString (int start, int end)
String	toString ()
String	toString (String programName)
String	toString (int start, int end)
String	toString (String programName, int start, int end)
String	toDebugString ()
String	toDebugString (int start, int end)
int	getLastRewriteTokenIndex ()
int	size ()
int	index ()
String	getEntireText ()
TokenOffsetInfo	getOffsetInfo (Token token)
Static Public Attributes
static final int	MIN_TOKEN_INDEX = 0
static final String	DEFAULT_PROGRAM_NAME = "default"
static final int	PROGRAM_INIT_SIZE = 100
Protected Member Functions
void	addToSortedRewriteList (RewriteOperation op)
void	addToSortedRewriteList (String programName, RewriteOperation op)
int	getLastRewriteTokenIndex (String programName)
void	setLastRewriteTokenIndex (String programName, int i)
List	getProgram (String name)
Protected Attributes
List	tokens
Map	programs = null
Map	lastRewriteTokenIndexes = null
int	index = MIN_TOKEN_INDEX
TokenStream	stream
BitSet	discardMask = new BitSet()

Detailed Description

This token stream tracks the *entire* token stream coming from a lexer, but does not pass on the whitespace (or whatever else you want to discard) to the parser.

This class can then be asked for the ith token in the input stream. Useful for dumping out the input stream exactly after doing some augmentation or other manipulations. Tokens are index from 0..n-1

You can insert stuff, replace, and delete chunks. Note that the operations are done lazily--only if you convert the buffer to a String. This is very efficient because you are not moving data around all the time. As the buffer of tokens is converted to strings, the toString() method(s) check to see if there is an operation at the current index. If so, the operation is done and then normal String rendering continues on the buffer. This is like having multiple Turing machine instruction streams (programs) operating on a single input tape. :)

Since the operations are done lazily at toString-time, operations do not screw up the token index values. That is, an insert operation at token index i does not change the index values for tokens i+1..n-1.

Because operations never actually alter the buffer, you may always get the original token stream back without undoing anything. Since the instructions are queued up, you can easily simulate transactions and roll back any changes if there is an error just by removing instructions. For example,

TokenStreamRewriteEngine rewriteEngine = new TokenStreamRewriteEngine(lexer); JavaRecognizer parser = new JavaRecognizer(rewriteEngine); ... rewriteEngine.insertAfter("pass1", t, "foobar");} rewriteEngine.insertAfter("pass2", u, "start");} System.out.println(rewriteEngine.toString("pass1")); System.out.println(rewriteEngine.toString("pass2"));

You can also have multiple "instruction streams" and get multiple rewrites from a single pass over the input. Just name the instruction streams and use that name again when printing the buffer. This could be useful for generating a C file and also its header file--all from the same buffer.

If you don't use named rewrite streams, a "default" stream is used.

Terence Parr, parrt at antlr.org University of San Francisco February 2004

Constructor & Destructor Documentation

antlr.TokenStreamRewriteEngine.TokenStreamRewriteEngine ( TokenStream upstream )

antlr.TokenStreamRewriteEngine.TokenStreamRewriteEngine	(	TokenStream	upstream,
		int	initialSize
	)

References antlr.TokenStreamRewriteEngine.DEFAULT_PROGRAM_NAME, antlr.TokenStreamRewriteEngine.lastRewriteTokenIndexes, antlr.TokenStreamRewriteEngine.PROGRAM_INIT_SIZE, antlr.TokenStreamRewriteEngine.programs, antlr.TokenStreamRewriteEngine.stream, and antlr.TokenStreamRewriteEngine.tokens.

Member Function Documentation

void antlr.TokenStreamRewriteEngine.addToSortedRewriteList ( RewriteOperation op ) [protected]

If op.index > lastRewriteTokenIndexes, just add to the end. Otherwise, do linear

References antlr.TokenStreamRewriteEngine.DEFAULT_PROGRAM_NAME.

Referenced by antlr.TokenStreamRewriteEngine.insertBefore(), and antlr.TokenStreamRewriteEngine.replace().

void antlr.TokenStreamRewriteEngine.addToSortedRewriteList	(	String	programName,
		RewriteOperation	op
	)			`[protected]`

old; before moving v3 stuff in protected void addToSortedRewriteList(String programName, RewriteOperation op) { List rewrites = getProgram(programName); if at or beyond last op's index, just append if ( op.index>=getLastRewriteTokenIndex(programName) ) { rewrites.add(op); // append to list of operations record the index of this operation for next time through setLastRewriteTokenIndex(programName, op.index); return; } not after the last one, so must insert to ordered list Comparator comparator = new Comparator() { public int compare(Object o, Object o1) { RewriteOperation a = (RewriteOperation)o; RewriteOperation b = (RewriteOperation)o1; if ( a.index<b.index ) return -1; if ( a.index>b.index ) return 1; return 0; } }; int pos = Collections.binarySearch(rewrites, op, comparator); if ( pos<0 ) { rewrites.add(-pos-1, op); } } Add an instruction to the rewrite instruction list ordered by the instruction number (use a binary search for efficiency). The list is ordered so that toString() can be done efficiently.

When there are multiple instructions at the same index, the instructions must be ordered to ensure proper behavior. For example, a delete at index i must kill any replace operation at i. Insert-before operations must come before any replace / delete instructions. If there are multiple insert instructions for a single index, they are done in reverse insertion order so that "insert foo" then "insert bar" yields "foobar" in front rather than "barfoo". This is convenient because I can insert new InsertOp instructions at the index returned by the binary search. A ReplaceOp kills any previous replace op. Since delete is the same as replace with null text, i can check for ReplaceOp and cover DeleteOp at same time. :)

References antlr.TokenStreamRewriteEngine.getProgram().

void antlr.TokenStreamRewriteEngine.delete ( int index )

References antlr.TokenStreamRewriteEngine.DEFAULT_PROGRAM_NAME, and antlr.TokenStreamRewriteEngine.index.

void antlr.TokenStreamRewriteEngine.delete	(	int	from,
		int	to
	)

References antlr.TokenStreamRewriteEngine.DEFAULT_PROGRAM_NAME.

void antlr.TokenStreamRewriteEngine.delete ( Token indexT )

References antlr.TokenStreamRewriteEngine.DEFAULT_PROGRAM_NAME.

void antlr.TokenStreamRewriteEngine.delete	(	Token	from,
		Token	to
	)

References antlr.TokenStreamRewriteEngine.DEFAULT_PROGRAM_NAME.

void antlr.TokenStreamRewriteEngine.delete	(	String	programName,
		int	from,
		int	to
	)

References antlr.TokenStreamRewriteEngine.replace().

void antlr.TokenStreamRewriteEngine.delete	(	String	programName,
		Token	from,
		Token	to
	)

References antlr.TokenStreamRewriteEngine.replace().

void antlr.TokenStreamRewriteEngine.deleteProgram ( String programName )

Reset the program so that no instructions exist

References antlr.TokenStreamRewriteEngine.MIN_TOKEN_INDEX, and antlr.TokenStreamRewriteEngine.rollback().

void antlr.TokenStreamRewriteEngine.deleteProgram ( )

References antlr.TokenStreamRewriteEngine.DEFAULT_PROGRAM_NAME.

void antlr.TokenStreamRewriteEngine.discard ( int ttype )

References antlr.collections.impl.BitSet.add(), and antlr.TokenStreamRewriteEngine.discardMask.

String antlr.TokenStreamRewriteEngine.getEntireText ( )

Returns the entire text input to the lexer.

Returns:: The entire text or null, if error occured or System.in was used.

Implements antlr.ASdebug.IASDebugStream.

References antlr.TokenStreamRewriteEngine.stream.

int antlr.TokenStreamRewriteEngine.getLastRewriteTokenIndex ( )

References antlr.TokenStreamRewriteEngine.DEFAULT_PROGRAM_NAME.

int antlr.TokenStreamRewriteEngine.getLastRewriteTokenIndex ( String programName ) [protected]

References antlr.TokenStreamRewriteEngine.lastRewriteTokenIndexes.

TokenOffsetInfo antlr.TokenStreamRewriteEngine.getOffsetInfo ( Token token )

Returns the offset information for the token

Parameters:

token

the token whose information need to be retrieved

Returns:: offset info, or null

Implements antlr.ASdebug.IASDebugStream.

References antlr.TokenStreamRewriteEngine.stream.

List antlr.TokenStreamRewriteEngine.getProgram ( String name ) [protected]

References antlr.TokenStreamRewriteEngine.programs.

Referenced by antlr.TokenStreamRewriteEngine.addToSortedRewriteList().

TokenWithIndex antlr.TokenStreamRewriteEngine.getToken ( int i )

References antlr.TokenStreamRewriteEngine.tokens.

Referenced by antlr.TokenStreamRewriteEngine.toDebugString(), antlr.TokenStreamRewriteEngine.toOriginalString(), and antlr.TokenStreamRewriteEngine.toString().

int antlr.TokenStreamRewriteEngine.getTokenStreamSize ( )

References antlr.TokenStreamRewriteEngine.tokens.

Referenced by antlr.TokenStreamRewriteEngine.toDebugString(), antlr.TokenStreamRewriteEngine.toOriginalString(), and antlr.TokenStreamRewriteEngine.toString().

int antlr.TokenStreamRewriteEngine.index ( )

References antlr.TokenStreamRewriteEngine.index.

Referenced by antlr.TokenStreamRewriteEngine.nextToken().

void antlr.TokenStreamRewriteEngine.insertAfter	(	Token	t,
		String	text
	)

References antlr.TokenStreamRewriteEngine.DEFAULT_PROGRAM_NAME.

Referenced by antlr.TokenStreamRewriteEngine.insertAfter().

void antlr.TokenStreamRewriteEngine.insertAfter	(	int	index,
		String	text
	)

References antlr.TokenStreamRewriteEngine.DEFAULT_PROGRAM_NAME, and antlr.TokenStreamRewriteEngine.insertAfter().

void antlr.TokenStreamRewriteEngine.insertAfter	(	String	programName,
		int	index,
		String	text
	)

References antlr.TokenStreamRewriteEngine.insertBefore().

void antlr.TokenStreamRewriteEngine.insertAfter	(	String	programName,
		Token	t,
		String	text
	)

References antlr.TokenStreamRewriteEngine.insertAfter().

void antlr.TokenStreamRewriteEngine.insertBefore	(	Token	t,
		String	text
	)

References antlr.TokenStreamRewriteEngine.DEFAULT_PROGRAM_NAME.

Referenced by antlr.TokenStreamRewriteEngine.insertAfter(), and antlr.TokenStreamRewriteEngine.insertBefore().

void antlr.TokenStreamRewriteEngine.insertBefore	(	int	index,
		String	text
	)

References antlr.TokenStreamRewriteEngine.DEFAULT_PROGRAM_NAME, and antlr.TokenStreamRewriteEngine.insertBefore().

void antlr.TokenStreamRewriteEngine.insertBefore	(	String	programName,
		Token	t,
		String	text
	)

References antlr.TokenStreamRewriteEngine.insertBefore().

void antlr.TokenStreamRewriteEngine.insertBefore	(	String	programName,
		int	index,
		String	text
	)

References antlr.TokenStreamRewriteEngine.addToSortedRewriteList().

Token antlr.TokenStreamRewriteEngine.nextToken ( ) throws TokenStreamException

References antlr.TokenStreamRewriteEngine.discardMask, antlr.Token.getType(), antlr.TokenStreamRewriteEngine.index(), antlr.collections.impl.BitSet.member(), antlr.TokenStream.nextToken(), antlr.TokenWithIndex.setIndex(), antlr.TokenStreamRewriteEngine.stream, and antlr.TokenStreamRewriteEngine.tokens.

void antlr.TokenStreamRewriteEngine.replace	(	int	index,
		String	text
	)

References antlr.TokenStreamRewriteEngine.DEFAULT_PROGRAM_NAME.

Referenced by antlr.TokenStreamRewriteEngine.delete(), and antlr.TokenStreamRewriteEngine.replace().

void antlr.TokenStreamRewriteEngine.replace	(	int	from,
		int	to,
		String	text
	)

References antlr.TokenStreamRewriteEngine.DEFAULT_PROGRAM_NAME, and antlr.TokenStreamRewriteEngine.replace().

void antlr.TokenStreamRewriteEngine.replace	(	Token	indexT,
		String	text
	)

References antlr.TokenStreamRewriteEngine.DEFAULT_PROGRAM_NAME, and antlr.TokenStreamRewriteEngine.replace().

void antlr.TokenStreamRewriteEngine.replace	(	String	programName,
		Token	from,
		Token	to,
		String	text
	)

References antlr.TokenStreamRewriteEngine.replace().

void antlr.TokenStreamRewriteEngine.replace	(	Token	from,
		Token	to,
		String	text
	)

References antlr.TokenStreamRewriteEngine.DEFAULT_PROGRAM_NAME, and antlr.TokenStreamRewriteEngine.replace().

void antlr.TokenStreamRewriteEngine.replace	(	String	programName,
		int	from,
		int	to,
		String	text
	)

References antlr.TokenStreamRewriteEngine.addToSortedRewriteList().

void antlr.TokenStreamRewriteEngine.rollback ( int instructionIndex )

References antlr.TokenStreamRewriteEngine.DEFAULT_PROGRAM_NAME.

Referenced by antlr.TokenStreamRewriteEngine.deleteProgram().

void antlr.TokenStreamRewriteEngine.rollback	(	String	programName,
		int	instructionIndex
	)

Rollback the instruction stream for a program so that the indicated instruction (via instructionIndex) is no longer in the stream. UNTESTED!

References antlr.TokenStreamRewriteEngine.MIN_TOKEN_INDEX, and antlr.TokenStreamRewriteEngine.programs.

void antlr.TokenStreamRewriteEngine.setLastRewriteTokenIndex	(	String	programName,
		int	i
	)			`[protected]`

References antlr.TokenStreamRewriteEngine.lastRewriteTokenIndexes.

int antlr.TokenStreamRewriteEngine.size ( )

References antlr.TokenStreamRewriteEngine.tokens.

Referenced by antlr.TokenStreamRewriteEngine.toString().

String antlr.TokenStreamRewriteEngine.toDebugString	(	int	start,
		int	end
	)

References antlr.TokenStreamRewriteEngine.getToken(), antlr.TokenStreamRewriteEngine.MIN_TOKEN_INDEX, and antlr.TokenStreamRewriteEngine.tokens.

String antlr.TokenStreamRewriteEngine.toDebugString ( )

References antlr.TokenStreamRewriteEngine.getTokenStreamSize(), and antlr.TokenStreamRewriteEngine.MIN_TOKEN_INDEX.

String antlr.TokenStreamRewriteEngine.toOriginalString ( )

References antlr.TokenStreamRewriteEngine.getTokenStreamSize(), and antlr.TokenStreamRewriteEngine.MIN_TOKEN_INDEX.

Referenced by antlr.TokenStreamRewriteEngine.toString().

String antlr.TokenStreamRewriteEngine.toOriginalString	(	int	start,
		int	end
	)

References antlr.TokenStreamRewriteEngine.getToken(), antlr.TokenStreamRewriteEngine.MIN_TOKEN_INDEX, and antlr.TokenStreamRewriteEngine.tokens.

String antlr.TokenStreamRewriteEngine.toString ( String programName )

References antlr.TokenStreamRewriteEngine.getTokenStreamSize(), antlr.TokenStreamRewriteEngine.MIN_TOKEN_INDEX, and antlr.TokenStreamRewriteEngine.toString().

String antlr.TokenStreamRewriteEngine.toString	(	int	start,
		int	end
	)

References antlr.TokenStreamRewriteEngine.DEFAULT_PROGRAM_NAME, and antlr.TokenStreamRewriteEngine.toString().

String antlr.TokenStreamRewriteEngine.toString ( )

References antlr.TokenStreamRewriteEngine.getTokenStreamSize(), and antlr.TokenStreamRewriteEngine.MIN_TOKEN_INDEX.

Referenced by antlr.TokenStreamRewriteEngine.toString().

String antlr.TokenStreamRewriteEngine.toString	(	String	programName,
		int	start,
		int	end
	)

Index of first rewrite we have not done

References antlr.TokenStreamRewriteEngine.getToken(), antlr.TokenStreamRewriteEngine.MIN_TOKEN_INDEX, antlr.TokenStreamRewriteEngine.programs, antlr.TokenStreamRewriteEngine.size(), antlr.TokenStreamRewriteEngine.tokens, and antlr.TokenStreamRewriteEngine.toOriginalString().

Member Data Documentation

final String antlr.TokenStreamRewriteEngine.DEFAULT_PROGRAM_NAME = "default" [static]

BitSet antlr.TokenStreamRewriteEngine.discardMask = new BitSet() [protected]

Which (whitespace) token(s) to throw out

Referenced by antlr.TokenStreamRewriteEngine.discard(), and antlr.TokenStreamRewriteEngine.nextToken().

int antlr.TokenStreamRewriteEngine.index = MIN_TOKEN_INDEX [protected]

track index of tokens

Referenced by antlr.TokenStreamRewriteEngine.delete(), and antlr.TokenStreamRewriteEngine.index().

Map antlr.TokenStreamRewriteEngine.lastRewriteTokenIndexes = null [protected]

Map String (program name) -> Integer index

Referenced by antlr.TokenStreamRewriteEngine.getLastRewriteTokenIndex(), antlr.TokenStreamRewriteEngine.setLastRewriteTokenIndex(), and antlr.TokenStreamRewriteEngine.TokenStreamRewriteEngine().

final int antlr.TokenStreamRewriteEngine.MIN_TOKEN_INDEX = 0 [static]

Referenced by antlr.TokenStreamRewriteEngine.deleteProgram(), antlr.TokenStreamRewriteEngine.rollback(), antlr.TokenStreamRewriteEngine.toDebugString(), antlr.TokenStreamRewriteEngine.toOriginalString(), and antlr.TokenStreamRewriteEngine.toString().

final int antlr.TokenStreamRewriteEngine.PROGRAM_INIT_SIZE = 100 [static]

Referenced by antlr.TokenStreamRewriteEngine.TokenStreamRewriteEngine().

Map antlr.TokenStreamRewriteEngine.programs = null [protected]

You may have multiple, named streams of rewrite operations. I'm calling these things "programs." Maps String (name) -> rewrite (List)

Referenced by antlr.TokenStreamRewriteEngine.getProgram(), antlr.TokenStreamRewriteEngine.rollback(), antlr.TokenStreamRewriteEngine.TokenStreamRewriteEngine(), and antlr.TokenStreamRewriteEngine.toString().

TokenStream antlr.TokenStreamRewriteEngine.stream [protected]

Who do we suck tokens from?

Referenced by antlr.TokenStreamRewriteEngine.getEntireText(), antlr.TokenStreamRewriteEngine.getOffsetInfo(), antlr.TokenStreamRewriteEngine.nextToken(), and antlr.TokenStreamRewriteEngine.TokenStreamRewriteEngine().

List antlr.TokenStreamRewriteEngine.tokens [protected]

Track the incoming list of tokens

Referenced by antlr.TokenStreamRewriteEngine.getToken(), antlr.TokenStreamRewriteEngine.getTokenStreamSize(), antlr.TokenStreamRewriteEngine.nextToken(), antlr.TokenStreamRewriteEngine.size(), antlr.TokenStreamRewriteEngine.toDebugString(), antlr.TokenStreamRewriteEngine.TokenStreamRewriteEngine(), antlr.TokenStreamRewriteEngine.toOriginalString(), and antlr.TokenStreamRewriteEngine.toString().

The documentation for this class was generated from the following file:

TokenStreamRewriteEngine.java

All Classes Namespaces Files Functions Variables Enumerations Properties

antlr.TokenStreamRewriteEngine Class Reference

Classes

Public Member Functions

Static Public Attributes

Protected Member Functions

Protected Attributes

Detailed Description

Constructor & Destructor Documentation

Member Function Documentation

Member Data Documentation