Package org.antlr.runtime
Class TokenRewriteStream
java.lang.Object
org.antlr.runtime.BufferedTokenStream
org.antlr.runtime.CommonTokenStream
org.antlr.runtime.TokenRewriteStream
- All Implemented Interfaces:
IntStream
,TokenStream
Useful for dumping out the input stream after doing some
augmentation or other manipulations.
You can insert stuff, replace, and delete chunks. Note that the
operations are done lazily--only if you convert the buffer to a
String. This is very efficient because you are not moving data around
all the time. As the buffer of tokens is converted to strings, the
toString() method(s) check to see if there is an operation at the
current index. If so, the operation is done and then normal String
rendering continues on the buffer. This is like having multiple Turing
machine instruction streams (programs) operating on a single input tape. :)
Since the operations are done lazily at toString-time, operations do not
screw up the token index values. That is, an insert operation at token
index i does not change the index values for tokens i+1..n-1.
Because operations never actually alter the buffer, you may always get
the original token stream back without undoing anything. Since
the instructions are queued up, you can easily simulate transactions and
roll back any changes if there is an error just by removing instructions.
For example,
CharStream input = new ANTLRFileStream("input");
TLexer lex = new TLexer(input);
TokenRewriteStream tokens = new TokenRewriteStream(lex);
T parser = new T(tokens);
parser.startRule();
Then in the rules, you can execute
Token t,u;
...
input.insertAfter(t, "text to put after t");}
input.insertAfter(u, "text after u");}
System.out.println(tokens.toString());
Actually, you have to cast the 'input' to a TokenRewriteStream. :(
You can also have multiple "instruction streams" and get multiple
rewrites from a single pass over the input. Just name the instruction
streams and use that name again when printing the buffer. This could be
useful for generating a C file and also its header file--all from the
same buffer:
tokens.insertAfter("pass1", t, "text to put after t");}
tokens.insertAfter("pass2", u, "text after u");}
System.out.println(tokens.toString("pass1"));
System.out.println(tokens.toString("pass2"));
If you don't use named rewrite streams, a "default" stream is used as
the first example shows.
-
Nested Class Summary
Nested ClassesModifier and TypeClassDescription(package private) class
(package private) class
I'm going to try replacing range from x..y with (y-x)+1 ReplaceOp instructions.class
-
Field Summary
FieldsModifier and TypeFieldDescriptionstatic final String
Map String (program name) → Integer indexstatic final int
static final int
protected Map<String,
List<TokenRewriteStream.RewriteOperation>> You may have multiple, named streams of rewrite operations.Fields inherited from class org.antlr.runtime.CommonTokenStream
channel
Fields inherited from class org.antlr.runtime.BufferedTokenStream
lastMarker, p, range, tokens, tokenSource
-
Constructor Summary
ConstructorsConstructorDescriptionTokenRewriteStream
(TokenSource tokenSource) TokenRewriteStream
(TokenSource tokenSource, int channel) -
Method Summary
Modifier and TypeMethodDescriptionprotected String
void
delete
(int index) void
delete
(int from, int to) void
void
void
void
void
void
deleteProgram
(String programName) Reset the program so that no instructions existprotected <T extends TokenRewriteStream.RewriteOperation>
List<? extends T>getKindOfOps
(List<? extends TokenRewriteStream.RewriteOperation> rewrites, Class<T> kind) protected <T extends TokenRewriteStream.RewriteOperation>
List<? extends T>getKindOfOps
(List<? extends TokenRewriteStream.RewriteOperation> rewrites, Class<T> kind, int before) Get all operations before an index of a particular kindint
protected int
getLastRewriteTokenIndex
(String programName) protected List<TokenRewriteStream.RewriteOperation>
getProgram
(String name) protected void
init()
private List<TokenRewriteStream.RewriteOperation>
initializeProgram
(String name) void
insertAfter
(int index, Object text) void
insertAfter
(String programName, int index, Object text) void
insertAfter
(String programName, Token t, Object text) void
insertAfter
(Token t, Object text) void
insertBefore
(int index, Object text) void
insertBefore
(String programName, int index, Object text) void
insertBefore
(String programName, Token t, Object text) void
insertBefore
(Token t, Object text) protected Map<Integer,
? extends TokenRewriteStream.RewriteOperation> reduceToSingleOperationPerIndex
(List<? extends TokenRewriteStream.RewriteOperation> rewrites) We need to combine operations and report invalid operations (like overlapping replaces that are not completed nested).void
void
void
void
void
void
void
rollback
(int instructionIndex) void
Rollback the instruction stream for a program so that the indicated instruction (via instructionIndex) is no longer in the stream.protected void
setLastRewriteTokenIndex
(String programName, int i) toDebugString
(int start, int end) toOriginalString
(int start, int end) toString()
Grab *all* tokens from stream and return stringtoString
(int start, int end) Return the text of all tokens from start to stop, inclusive.Methods inherited from class org.antlr.runtime.CommonTokenStream
consume, getNumberOfOnChannelTokens, LB, LT, reset, setTokenSource, setup, skipOffTokenChannels, skipOffTokenChannelsReverse
-
Field Details
-
DEFAULT_PROGRAM_NAME
- See Also:
-
PROGRAM_INIT_SIZE
public static final int PROGRAM_INIT_SIZE- See Also:
-
MIN_TOKEN_INDEX
public static final int MIN_TOKEN_INDEX- See Also:
-
programs
You may have multiple, named streams of rewrite operations. I'm calling these things "programs." Maps String (name) → rewrite (List) -
lastRewriteTokenIndexes
Map String (program name) → Integer index
-
-
Constructor Details
-
TokenRewriteStream
public TokenRewriteStream() -
TokenRewriteStream
-
TokenRewriteStream
-
-
Method Details
-
init
protected void init() -
rollback
public void rollback(int instructionIndex) -
rollback
Rollback the instruction stream for a program so that the indicated instruction (via instructionIndex) is no longer in the stream. UNTESTED! -
deleteProgram
public void deleteProgram() -
deleteProgram
Reset the program so that no instructions exist -
insertAfter
-
insertAfter
-
insertAfter
-
insertAfter
-
insertBefore
-
insertBefore
-
insertBefore
-
insertBefore
-
replace
-
replace
-
replace
-
replace
-
replace
-
replace
-
delete
public void delete(int index) -
delete
public void delete(int from, int to) -
delete
-
delete
-
delete
-
delete
-
getLastRewriteTokenIndex
public int getLastRewriteTokenIndex() -
getLastRewriteTokenIndex
-
setLastRewriteTokenIndex
-
getProgram
-
initializeProgram
-
toOriginalString
-
toOriginalString
-
toString
Description copied from class:BufferedTokenStream
Grab *all* tokens from stream and return string- Overrides:
toString
in classBufferedTokenStream
-
toString
-
toString
Description copied from interface:TokenStream
Return the text of all tokens from start to stop, inclusive. If the stream does not buffer all the tokens then it can just return "" or null; Users should not access $ruleLabel.text in an action of course in that case.- Specified by:
toString
in interfaceTokenStream
- Overrides:
toString
in classBufferedTokenStream
-
toString
-
reduceToSingleOperationPerIndex
protected Map<Integer,? extends TokenRewriteStream.RewriteOperation> reduceToSingleOperationPerIndex(List<? extends TokenRewriteStream.RewriteOperation> rewrites) We need to combine operations and report invalid operations (like overlapping replaces that are not completed nested). Inserts to same index need to be combined etc... Here are the cases: I.i.u I.j.v leave alone, nonoverlapping I.i.u I.i.v combine: Iivu R.i-j.u R.x-y.v | i-j in x-y delete first R R.i-j.u R.i-j.v delete first R R.i-j.u R.x-y.v | x-y in i-j ERROR R.i-j.u R.x-y.v | boundaries overlap ERROR Delete special case of replace (text==null): D.i-j.u D.x-y.v | boundaries overlap combine to max(min)..max(right) I.i.u R.x-y.v | i in (x+1)-y delete I (since insert before we're not deleting i) I.i.u R.x-y.v | i not in (x+1)-y leave alone, nonoverlapping R.x-y.v I.i.u | i in x-y ERROR R.x-y.v I.x.u R.x-y.uv (combine, delete I) R.x-y.v I.i.u | i not in x-y leave alone, nonoverlapping I.i.u = insert u before op @ index i R.x-y.u = replace x-y indexed tokens with u First we need to examine replaces. For any replace op: 1. wipe out any insertions before op within that range. 2. Drop any replace op before that is contained completely within that range. 3. Throw exception upon boundary overlap with any previous replace. Then we can deal with inserts: 1. for any inserts to same index, combine even if not adjacent. 2. for any prior replace with same left boundary, combine this insert with replace and delete this replace. 3. throw exception if index in same range as previous replace Don't actually delete; make op null in list. Easier to walk list. Later we can throw as we add to index → op map. Note that I.2 R.2-2 will wipe out I.2 even though, technically, the inserted stuff would be before the replace range. But, if you add tokens in front of a method body '{' and then delete the method body, I think the stuff before the '{' you added should disappear too. Return a map from token index to operation. -
catOpText
-
getKindOfOps
protected <T extends TokenRewriteStream.RewriteOperation> List<? extends T> getKindOfOps(List<? extends TokenRewriteStream.RewriteOperation> rewrites, Class<T> kind) -
getKindOfOps
protected <T extends TokenRewriteStream.RewriteOperation> List<? extends T> getKindOfOps(List<? extends TokenRewriteStream.RewriteOperation> rewrites, Class<T> kind, int before) Get all operations before an index of a particular kind -
toDebugString
-
toDebugString
-