Class RetokenizingTokenSource

  extended by
All Implemented Interfaces:

public class RetokenizingTokenSource
extends java.lang.Object
implements org.antlr.runtime.TokenSource

RetokenizingTokenSource accomodates backtracking limitations in the ANTRL lexer when used from Xtext. The problem is the three overlapping syntaxes INT.INT leading to a Floating Point lteral INT..INT leading to a Collection range INT.ID leading to a numeric navigation ANTLR proceeds to INT. but won't backup when the character after the . is bad. The code here allows the basic lexer to be ignorant of floating point syntax so that it correctly parses INT.INT as INT DOT INT INT..INT as INT DOTDOT INT INT.INTe+INT as INT DOT INT ID PLUS INT so the code here recognises the floating point literal and reconsttructs. As an additional benefit 'e' and 'E' are not keywords.

Field Summary
protected  java.util.LinkedList<org.antlr.runtime.Token> queue
protected  java.util.Map<java.lang.Integer,java.lang.String> tokenDefMap
protected  org.antlr.runtime.TokenSource tokenSource
Constructor Summary
RetokenizingTokenSource(org.antlr.runtime.TokenSource tokenSource, java.util.Map<java.lang.Integer,java.lang.String> tokenDefMap)
Method Summary
 java.lang.String getSourceName()
protected  boolean isInteger(java.lang.String text)
 org.antlr.runtime.Token nextToken()
protected  org.antlr.runtime.Token peek(int index)
          Return the token index positions ahead of the next token.
Methods inherited from class java.lang.Object
clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Detail


protected final org.antlr.runtime.TokenSource tokenSource


protected final java.util.Map<java.lang.Integer,java.lang.String> tokenDefMap


protected final java.util.LinkedList<org.antlr.runtime.Token> queue
Constructor Detail


public RetokenizingTokenSource(org.antlr.runtime.TokenSource tokenSource,
                               java.util.Map<java.lang.Integer,java.lang.String> tokenDefMap)
Method Detail


public java.lang.String getSourceName()
Specified by:
getSourceName in interface org.antlr.runtime.TokenSource


protected boolean isInteger(java.lang.String text)


public org.antlr.runtime.Token nextToken()
Specified by:
nextToken in interface org.antlr.runtime.TokenSource


protected org.antlr.runtime.Token peek(int index)
Return the token index positions ahead of the next token. Returns EOF_TOKEN if no such token or if a hidden token interleaves.