Classes | |
class | DeleteOp |
class | InsertBeforeOp |
class | ReplaceOp |
class | RewriteOperation |
Public Member Functions | |
TokenStreamRewriteEngine (TokenStream upstream) | |
TokenStreamRewriteEngine (TokenStream upstream, int initialSize) | |
Token | nextToken () throws TokenStreamException |
void | rollback (int instructionIndex) |
void | rollback (String programName, int instructionIndex) |
void | deleteProgram () |
void | deleteProgram (String programName) |
void | insertAfter (Token t, String text) |
void | insertAfter (int index, String text) |
void | insertAfter (String programName, Token t, String text) |
void | insertAfter (String programName, int index, String text) |
void | insertBefore (Token t, String text) |
void | insertBefore (int index, String text) |
void | insertBefore (String programName, Token t, String text) |
void | insertBefore (String programName, int index, String text) |
void | replace (int index, String text) |
void | replace (int from, int to, String text) |
void | replace (Token indexT, String text) |
void | replace (Token from, Token to, String text) |
void | replace (String programName, int from, int to, String text) |
void | replace (String programName, Token from, Token to, String text) |
void | delete (int index) |
void | delete (int from, int to) |
void | delete (Token indexT) |
void | delete (Token from, Token to) |
void | delete (String programName, int from, int to) |
void | delete (String programName, Token from, Token to) |
void | discard (int ttype) |
TokenWithIndex | getToken (int i) |
int | getTokenStreamSize () |
String | toOriginalString () |
String | toOriginalString (int start, int end) |
String | toString () |
String | toString (String programName) |
String | toString (int start, int end) |
String | toString (String programName, int start, int end) |
String | toDebugString () |
String | toDebugString (int start, int end) |
int | getLastRewriteTokenIndex () |
int | size () |
int | index () |
String | getEntireText () |
TokenOffsetInfo | getOffsetInfo (Token token) |
Static Public Attributes | |
static final int | MIN_TOKEN_INDEX = 0 |
static final String | DEFAULT_PROGRAM_NAME = "default" |
static final int | PROGRAM_INIT_SIZE = 100 |
Protected Member Functions | |
void | addToSortedRewriteList (RewriteOperation op) |
void | addToSortedRewriteList (String programName, RewriteOperation op) |
int | getLastRewriteTokenIndex (String programName) |
void | setLastRewriteTokenIndex (String programName, int i) |
List | getProgram (String name) |
Protected Attributes | |
List | tokens |
Map | programs = null |
Map | lastRewriteTokenIndexes = null |
int | index = MIN_TOKEN_INDEX |
TokenStream | stream |
BitSet | discardMask = new BitSet() |
This token stream tracks the *entire* token stream coming from a lexer, but does not pass on the whitespace (or whatever else you want to discard) to the parser.
This class can then be asked for the ith token in the input stream. Useful for dumping out the input stream exactly after doing some augmentation or other manipulations. Tokens are index from 0..n-1
You can insert stuff, replace, and delete chunks. Note that the operations are done lazily--only if you convert the buffer to a String. This is very efficient because you are not moving data around all the time. As the buffer of tokens is converted to strings, the toString() method(s) check to see if there is an operation at the current index. If so, the operation is done and then normal String rendering continues on the buffer. This is like having multiple Turing machine instruction streams (programs) operating on a single input tape. :)
Since the operations are done lazily at toString-time, operations do not screw up the token index values. That is, an insert operation at token index i does not change the index values for tokens i+1..n-1.
Because operations never actually alter the buffer, you may always get the original token stream back without undoing anything. Since the instructions are queued up, you can easily simulate transactions and roll back any changes if there is an error just by removing instructions. For example,
TokenStreamRewriteEngine rewriteEngine = new TokenStreamRewriteEngine(lexer); JavaRecognizer parser = new JavaRecognizer(rewriteEngine); ... rewriteEngine.insertAfter("pass1", t, "foobar");} rewriteEngine.insertAfter("pass2", u, "start");} System.out.println(rewriteEngine.toString("pass1")); System.out.println(rewriteEngine.toString("pass2"));
You can also have multiple "instruction streams" and get multiple rewrites from a single pass over the input. Just name the instruction streams and use that name again when printing the buffer. This could be useful for generating a C file and also its header file--all from the same buffer.
If you don't use named rewrite streams, a "default" stream is used.
Terence Parr, parrt at antlr.org University of San Francisco February 2004
antlr.TokenStreamRewriteEngine.TokenStreamRewriteEngine | ( | TokenStream | upstream | ) |
antlr.TokenStreamRewriteEngine.TokenStreamRewriteEngine | ( | TokenStream | upstream, | |
int | initialSize | |||
) |
void antlr.TokenStreamRewriteEngine.addToSortedRewriteList | ( | RewriteOperation | op | ) | [protected] |
If op.index > lastRewriteTokenIndexes, just add to the end. Otherwise, do linear
References antlr.TokenStreamRewriteEngine.DEFAULT_PROGRAM_NAME.
Referenced by antlr.TokenStreamRewriteEngine.insertBefore(), and antlr.TokenStreamRewriteEngine.replace().
void antlr.TokenStreamRewriteEngine.addToSortedRewriteList | ( | String | programName, | |
RewriteOperation | op | |||
) | [protected] |
old; before moving v3 stuff in protected void addToSortedRewriteList(String programName, RewriteOperation op) { List rewrites = getProgram(programName); if at or beyond last op's index, just append if ( op.index>=getLastRewriteTokenIndex(programName) ) { rewrites.add(op); // append to list of operations record the index of this operation for next time through setLastRewriteTokenIndex(programName, op.index); return; } not after the last one, so must insert to ordered list Comparator comparator = new Comparator() { public int compare(Object o, Object o1) { RewriteOperation a = (RewriteOperation)o; RewriteOperation b = (RewriteOperation)o1; if ( a.index<b.index ) return -1; if ( a.index>b.index ) return 1; return 0; } }; int pos = Collections.binarySearch(rewrites, op, comparator); if ( pos<0 ) { rewrites.add(-pos-1, op); } } Add an instruction to the rewrite instruction list ordered by the instruction number (use a binary search for efficiency). The list is ordered so that toString() can be done efficiently.
When there are multiple instructions at the same index, the instructions must be ordered to ensure proper behavior. For example, a delete at index i must kill any replace operation at i. Insert-before operations must come before any replace / delete instructions. If there are multiple insert instructions for a single index, they are done in reverse insertion order so that "insert foo" then "insert bar" yields "foobar" in front rather than "barfoo". This is convenient because I can insert new InsertOp instructions at the index returned by the binary search. A ReplaceOp kills any previous replace op. Since delete is the same as replace with null text, i can check for ReplaceOp and cover DeleteOp at same time. :)
References antlr.TokenStreamRewriteEngine.getProgram().
void antlr.TokenStreamRewriteEngine.delete | ( | int | index | ) |
void antlr.TokenStreamRewriteEngine.delete | ( | int | from, | |
int | to | |||
) |
void antlr.TokenStreamRewriteEngine.delete | ( | Token | indexT | ) |
void antlr.TokenStreamRewriteEngine.delete | ( | String | programName, | |
int | from, | |||
int | to | |||
) |
References antlr.TokenStreamRewriteEngine.replace().
References antlr.TokenStreamRewriteEngine.replace().
void antlr.TokenStreamRewriteEngine.deleteProgram | ( | String | programName | ) |
Reset the program so that no instructions exist
References antlr.TokenStreamRewriteEngine.MIN_TOKEN_INDEX, and antlr.TokenStreamRewriteEngine.rollback().
void antlr.TokenStreamRewriteEngine.deleteProgram | ( | ) |
void antlr.TokenStreamRewriteEngine.discard | ( | int | ttype | ) |
String antlr.TokenStreamRewriteEngine.getEntireText | ( | ) |
Returns the entire text input to the lexer.
null
, if error occured or System.in was used. Implements antlr.ASdebug.IASDebugStream.
References antlr.TokenStreamRewriteEngine.stream.
int antlr.TokenStreamRewriteEngine.getLastRewriteTokenIndex | ( | ) |
int antlr.TokenStreamRewriteEngine.getLastRewriteTokenIndex | ( | String | programName | ) | [protected] |
TokenOffsetInfo antlr.TokenStreamRewriteEngine.getOffsetInfo | ( | Token | token | ) |
Returns the offset information for the token
token | the token whose information need to be retrieved |
null
Implements antlr.ASdebug.IASDebugStream.
References antlr.TokenStreamRewriteEngine.stream.
List antlr.TokenStreamRewriteEngine.getProgram | ( | String | name | ) | [protected] |
References antlr.TokenStreamRewriteEngine.programs.
Referenced by antlr.TokenStreamRewriteEngine.addToSortedRewriteList().
TokenWithIndex antlr.TokenStreamRewriteEngine.getToken | ( | int | i | ) |
int antlr.TokenStreamRewriteEngine.getTokenStreamSize | ( | ) |
References antlr.TokenStreamRewriteEngine.index.
Referenced by antlr.TokenStreamRewriteEngine.nextToken().
void antlr.TokenStreamRewriteEngine.insertAfter | ( | Token | t, | |
String | text | |||
) |
References antlr.TokenStreamRewriteEngine.DEFAULT_PROGRAM_NAME.
Referenced by antlr.TokenStreamRewriteEngine.insertAfter().
void antlr.TokenStreamRewriteEngine.insertAfter | ( | int | index, | |
String | text | |||
) |
void antlr.TokenStreamRewriteEngine.insertAfter | ( | String | programName, | |
int | index, | |||
String | text | |||
) |
References antlr.TokenStreamRewriteEngine.insertBefore().
void antlr.TokenStreamRewriteEngine.insertAfter | ( | String | programName, | |
Token | t, | |||
String | text | |||
) |
References antlr.TokenStreamRewriteEngine.insertAfter().
void antlr.TokenStreamRewriteEngine.insertBefore | ( | Token | t, | |
String | text | |||
) |
void antlr.TokenStreamRewriteEngine.insertBefore | ( | int | index, | |
String | text | |||
) |
void antlr.TokenStreamRewriteEngine.insertBefore | ( | String | programName, | |
Token | t, | |||
String | text | |||
) |
References antlr.TokenStreamRewriteEngine.insertBefore().
void antlr.TokenStreamRewriteEngine.insertBefore | ( | String | programName, | |
int | index, | |||
String | text | |||
) |
Token antlr.TokenStreamRewriteEngine.nextToken | ( | ) | throws TokenStreamException |
References antlr.TokenStreamRewriteEngine.discardMask, antlr.Token.getType(), antlr.TokenStreamRewriteEngine.index(), antlr.collections.impl.BitSet.member(), antlr.TokenStream.nextToken(), antlr.TokenWithIndex.setIndex(), antlr.TokenStreamRewriteEngine.stream, and antlr.TokenStreamRewriteEngine.tokens.
void antlr.TokenStreamRewriteEngine.replace | ( | int | index, | |
String | text | |||
) |
References antlr.TokenStreamRewriteEngine.DEFAULT_PROGRAM_NAME.
Referenced by antlr.TokenStreamRewriteEngine.delete(), and antlr.TokenStreamRewriteEngine.replace().
void antlr.TokenStreamRewriteEngine.replace | ( | int | from, | |
int | to, | |||
String | text | |||
) |
void antlr.TokenStreamRewriteEngine.replace | ( | Token | indexT, | |
String | text | |||
) |
void antlr.TokenStreamRewriteEngine.replace | ( | String | programName, | |
Token | from, | |||
Token | to, | |||
String | text | |||
) |
References antlr.TokenStreamRewriteEngine.replace().
void antlr.TokenStreamRewriteEngine.replace | ( | String | programName, | |
int | from, | |||
int | to, | |||
String | text | |||
) |
void antlr.TokenStreamRewriteEngine.rollback | ( | int | instructionIndex | ) |
References antlr.TokenStreamRewriteEngine.DEFAULT_PROGRAM_NAME.
Referenced by antlr.TokenStreamRewriteEngine.deleteProgram().
void antlr.TokenStreamRewriteEngine.rollback | ( | String | programName, | |
int | instructionIndex | |||
) |
Rollback the instruction stream for a program so that the indicated instruction (via instructionIndex) is no longer in the stream. UNTESTED!
References antlr.TokenStreamRewriteEngine.MIN_TOKEN_INDEX, and antlr.TokenStreamRewriteEngine.programs.
void antlr.TokenStreamRewriteEngine.setLastRewriteTokenIndex | ( | String | programName, | |
int | i | |||
) | [protected] |
int antlr.TokenStreamRewriteEngine.size | ( | ) |
References antlr.TokenStreamRewriteEngine.tokens.
Referenced by antlr.TokenStreamRewriteEngine.toString().
String antlr.TokenStreamRewriteEngine.toDebugString | ( | int | start, | |
int | end | |||
) |
String antlr.TokenStreamRewriteEngine.toDebugString | ( | ) |
String antlr.TokenStreamRewriteEngine.toOriginalString | ( | ) |
String antlr.TokenStreamRewriteEngine.toOriginalString | ( | int | start, | |
int | end | |||
) |
String antlr.TokenStreamRewriteEngine.toString | ( | String | programName | ) |
String antlr.TokenStreamRewriteEngine.toString | ( | int | start, | |
int | end | |||
) |
String antlr.TokenStreamRewriteEngine.toString | ( | ) |
String antlr.TokenStreamRewriteEngine.toString | ( | String | programName, | |
int | start, | |||
int | end | |||
) |
Index of first rewrite we have not done
References antlr.TokenStreamRewriteEngine.getToken(), antlr.TokenStreamRewriteEngine.MIN_TOKEN_INDEX, antlr.TokenStreamRewriteEngine.programs, antlr.TokenStreamRewriteEngine.size(), antlr.TokenStreamRewriteEngine.tokens, and antlr.TokenStreamRewriteEngine.toOriginalString().
final String antlr.TokenStreamRewriteEngine.DEFAULT_PROGRAM_NAME = "default" [static] |
Referenced by antlr.TokenStreamRewriteEngine.addToSortedRewriteList(), antlr.TokenStreamRewriteEngine.delete(), antlr.TokenStreamRewriteEngine.deleteProgram(), antlr.TokenStreamRewriteEngine.getLastRewriteTokenIndex(), antlr.TokenStreamRewriteEngine.insertAfter(), antlr.TokenStreamRewriteEngine.insertBefore(), antlr.TokenStreamRewriteEngine.replace(), antlr.TokenStreamRewriteEngine.rollback(), antlr.TokenStreamRewriteEngine.TokenStreamRewriteEngine(), and antlr.TokenStreamRewriteEngine.toString().
BitSet antlr.TokenStreamRewriteEngine.discardMask = new BitSet() [protected] |
Which (whitespace) token(s) to throw out
Referenced by antlr.TokenStreamRewriteEngine.discard(), and antlr.TokenStreamRewriteEngine.nextToken().
int antlr.TokenStreamRewriteEngine.index = MIN_TOKEN_INDEX [protected] |
track index of tokens
Referenced by antlr.TokenStreamRewriteEngine.delete(), and antlr.TokenStreamRewriteEngine.index().
Map antlr.TokenStreamRewriteEngine.lastRewriteTokenIndexes = null [protected] |
Map String (program name) -> Integer index
Referenced by antlr.TokenStreamRewriteEngine.getLastRewriteTokenIndex(), antlr.TokenStreamRewriteEngine.setLastRewriteTokenIndex(), and antlr.TokenStreamRewriteEngine.TokenStreamRewriteEngine().
final int antlr.TokenStreamRewriteEngine.MIN_TOKEN_INDEX = 0 [static] |
final int antlr.TokenStreamRewriteEngine.PROGRAM_INIT_SIZE = 100 [static] |
Referenced by antlr.TokenStreamRewriteEngine.TokenStreamRewriteEngine().
Map antlr.TokenStreamRewriteEngine.programs = null [protected] |
You may have multiple, named streams of rewrite operations. I'm calling these things "programs." Maps String (name) -> rewrite (List)
Referenced by antlr.TokenStreamRewriteEngine.getProgram(), antlr.TokenStreamRewriteEngine.rollback(), antlr.TokenStreamRewriteEngine.TokenStreamRewriteEngine(), and antlr.TokenStreamRewriteEngine.toString().
TokenStream antlr.TokenStreamRewriteEngine.stream [protected] |
Who do we suck tokens from?
Referenced by antlr.TokenStreamRewriteEngine.getEntireText(), antlr.TokenStreamRewriteEngine.getOffsetInfo(), antlr.TokenStreamRewriteEngine.nextToken(), and antlr.TokenStreamRewriteEngine.TokenStreamRewriteEngine().
List antlr.TokenStreamRewriteEngine.tokens [protected] |
Track the incoming list of tokens
Referenced by antlr.TokenStreamRewriteEngine.getToken(), antlr.TokenStreamRewriteEngine.getTokenStreamSize(), antlr.TokenStreamRewriteEngine.nextToken(), antlr.TokenStreamRewriteEngine.size(), antlr.TokenStreamRewriteEngine.toDebugString(), antlr.TokenStreamRewriteEngine.TokenStreamRewriteEngine(), antlr.TokenStreamRewriteEngine.toOriginalString(), and antlr.TokenStreamRewriteEngine.toString().