TokenStreamRewriter

Useful for rewriting out a buffered input token stream after doing some augmentation or other manipulations on it.

You can insert stuff, replace, and delete chunks. Note that the operations are done lazily--only if you convert the buffer to a {@link String} with {@link TokenStream#getText()}. This is very efficient because you are not moving data around all the time. As the buffer of tokens is converted to strings, the {@link #getText()} method(s) scan the input token stream and check to see if there is an operation at the current index. If so, the operation is done and then normal {@link String} rendering continues on the buffer. This is like having multiple Turing machine instruction streams (programs) operating on a single input tape.

This rewriter makes no modifications to the token stream. It does not ask the stream to fill itself up nor does it advance the input cursor. The token stream {@link TokenStream#index()} will return the same value before and after any {@link #getText()} call.

The rewriter only works on tokens that you have in the buffer and ignores the current input cursor. If you are buffering tokens on-demand, calling {@link #getText()} halfway through the input will only do rewrites for those tokens in the first half of the file.

Since the operations are done lazily at {@link #getText}-time, operations do not screw up the token index values. That is, an insert operation at token index {@code i} does not change the index values for tokens {@code i}+1..n-1.

Because operations never actually alter the buffer, you may always get the original token stream back without undoing anything. Since the instructions are queued up, you can easily simulate transactions and roll back any changes if there is an error just by removing instructions. For example,

class TokenStreamRewriter {

enum string DEFAULT_PROGRAM_NAME;

enum int MIN_TOKEN_INDEX;

protected

RewriteOperation[][string] programs;

protected

size_t[string] lastRewriteTokenIndexes;

this();

this(TokenStream tokens);

void rollback(int instructionIndex);

void rollback(string programName, int instructionIndex);

void deleteProgram();

void deleteProgram(string programName);

void insertAfter(Token t, Variant text);

void insertAfter(int index, Variant text);

void insertAfter(string programName, Token t, Variant text);

void insertAfter(string programName, size_t index, Variant text);

void insertBefore(Token t, Variant text);

void insertBefore(size_t index, Variant text);

void insertBefore(string programName, Token t, Variant text);

void insertBefore(string programName, size_t index, Variant text);

void replace(size_t index, Variant text);

void replace(size_t from, size_t to, Variant text);

void replace(Token indexT, Variant text);

void replace(Token from, Token to, Variant text);

void replace(string programName, size_t from, size_t to, Variant text);

void replace(string programName, Token from, Token to, Variant text);

void deleteT(size_t index);

void deleteT(size_t from, size_t to);

void deleteT(Token indexT);

void deleteT(Token from, Token to);

void deleteT(string programName, size_t from, size_t to);

void deleteT(string programName, Token from, Token to);

size_t getLastRewriteTokenIndex();

Variant getText();

Variant getText(string programName);

Variant getText(Interval interval);

Variant getText(string programName, Interval interval);

RewriteOperation[size_t] reduceToSingleOperationPerIndex(RewriteOperation[] rewrites);

Variant catOpText(Variant a, Variant b);

auto getKindOfOps(RewriteOperation[] rewrites, size_t before);

static TokenStream tokens();

}

Constructors

this this(): Undocumented in source.
this this(TokenStream tokens): Undocumented in source.

Members

Functions

catOpText Variant catOpText(Variant a, Variant b): Undocumented in source. Be warned that the author may not have intended to support it.
deleteProgram void deleteProgram(): Undocumented in source. Be warned that the author may not have intended to support it.
deleteProgram void deleteProgram(string programName): Reset the program so that no instructions exist
deleteT void deleteT(size_t index): Delete token (can not use delete as identifier)
deleteT void deleteT(size_t from, size_t to): Undocumented in source. Be warned that the author may not have intended to support it.
deleteT void deleteT(Token indexT): Undocumented in source. Be warned that the author may not have intended to support it.
deleteT void deleteT(Token from, Token to): Undocumented in source. Be warned that the author may not have intended to support it.
deleteT void deleteT(string programName, size_t from, size_t to): Undocumented in source. Be warned that the author may not have intended to support it.
deleteT void deleteT(string programName, Token from, Token to): Undocumented in source. Be warned that the author may not have intended to support it.
getKindOfOps auto getKindOfOps(RewriteOperation[] rewrites, size_t before): Undocumented in source. Be warned that the author may not have intended to support it.
getLastRewriteTokenIndex size_t getLastRewriteTokenIndex(): Undocumented in source. Be warned that the author may not have intended to support it.
getText Variant getText(): Return the text from the original tokens altered per the instructions given to this rewriter.
getText Variant getText(string programName): Return the text from the original tokens altered per the instructions given to this rewriter in programName.
getText Variant getText(Interval interval): Return the text associated with the tokens in the interval from the original token stream but with the alterations given to this rewriter. The interval refers to the indexes in the original token stream. We do not alter the token stream in any way, so the indexes and intervals are still consistent. Includes any operations done to the first and last token in the interval. So, if you did an insertBefore on the first token, you would get that insertion. The same is true if you do an insertAfter the stop token.
getText Variant getText(string programName, Interval interval): Undocumented in source. Be warned that the author may not have intended to support it.
insertAfter void insertAfter(Token t, Variant text): Undocumented in source. Be warned that the author may not have intended to support it.
insertAfter void insertAfter(int index, Variant text): Undocumented in source. Be warned that the author may not have intended to support it.
insertAfter void insertAfter(string programName, Token t, Variant text): Undocumented in source. Be warned that the author may not have intended to support it.
insertAfter void insertAfter(string programName, size_t index, Variant text): Undocumented in source. Be warned that the author may not have intended to support it.
insertBefore void insertBefore(Token t, Variant text): Undocumented in source. Be warned that the author may not have intended to support it.
insertBefore void insertBefore(size_t index, Variant text): Undocumented in source. Be warned that the author may not have intended to support it.
insertBefore void insertBefore(string programName, Token t, Variant text): Undocumented in source. Be warned that the author may not have intended to support it.
insertBefore void insertBefore(string programName, size_t index, Variant text): Undocumented in source. Be warned that the author may not have intended to support it.
reduceToSingleOperationPerIndex RewriteOperation[size_t] reduceToSingleOperationPerIndex(RewriteOperation[] rewrites): We need to combine operations and report invalid operations (like overlapping replaces that are not completed nested). Inserts to same index need to be combined etc.
replace void replace(size_t index, Variant text): Undocumented in source. Be warned that the author may not have intended to support it.
replace void replace(size_t from, size_t to, Variant text): Undocumented in source. Be warned that the author may not have intended to support it.
replace void replace(Token indexT, Variant text): Undocumented in source. Be warned that the author may not have intended to support it.
replace void replace(Token from, Token to, Variant text): Undocumented in source. Be warned that the author may not have intended to support it.
replace void replace(string programName, size_t from, size_t to, Variant text): Undocumented in source. Be warned that the author may not have intended to support it.
replace void replace(string programName, Token from, Token to, Variant text): Undocumented in source. Be warned that the author may not have intended to support it.
rollback void rollback(int instructionIndex): Undocumented in source. Be warned that the author may not have intended to support it.
rollback void rollback(string programName, int instructionIndex): Rollback the instruction stream for a program so that the indicated instruction (via instructionIndex) is no longer in the stream. UNTESTED!

Static functions

tokens TokenStream tokens(): Undocumented in source. Be warned that the author may not have intended to support it.

Variables

DEFAULT_PROGRAM_NAME enum string DEFAULT_PROGRAM_NAME;: Undocumented in source.
MIN_TOKEN_INDEX enum int MIN_TOKEN_INDEX;: Undocumented in source.
lastRewriteTokenIndexes size_t[string] lastRewriteTokenIndexes;: Undocumented in source.
programs RewriteOperation[][string] programs;: Undocumented in source.

Examples

CharStream input = new ANTLRFileStream("input");
TLexer lex = new TLexer(input);
CommonTokenStream tokens = new CommonTokenStream(lex);
T parser = new T(tokens);
TokenStreamRewriter rewriter = new TokenStreamRewriter(tokens);
parser.startRule();

Then in the rules, you can execute (assuming rewriter is visible):

Token t,u;
...
rewriter.insertAfter(t, "text to put after t");}
rewriter.insertAfter(u, "text after u");}
System.out.println(rewriter.getText());

You can also have multiple "instruction streams" and get multiple rewrites from a single pass over the input. Just name the instruction streams and use that name again when printing the buffer. This could be useful for generating a C file and also its header file--all from the same buffer:

rewriter.insertAfter("pass1", t, "text to put after t");}
rewriter.insertAfter("pass2", u, "text after u");}
System.out.println(rewriter.getText("pass1"));
System.out.println(rewriter.getText("pass2"));

If you don't use named rewrite streams, a "default" stream is used as the first example shows.