org.eclipse.ocl.examples.xtext.essentialocl.services
Class RetokenizingTokenSource
java.lang.Object
org.eclipse.ocl.examples.xtext.essentialocl.services.RetokenizingTokenSource
- All Implemented Interfaces:
- org.antlr.runtime.TokenSource
public class RetokenizingTokenSource
- extends java.lang.Object
- implements org.antlr.runtime.TokenSource
RetokenizingTokenSource accomodates backtracking limitations in the ANTRL lexer when used from Xtext.
The problem is the three overlapping syntaxes
INT.INT leading to a Floating Point lteral
INT..INT leading to a Collection range
INT.ID leading to a numeric navigation
ANTLR proceeds to INT. but won't backup when the character after the . is bad.
The code here allows the basic lexer to be ignorant of floating point syntax so that it correctly parses
INT.INT as INT DOT INT
INT..INT as INT DOTDOT INT
INT.INTe+INT as INT DOT INT ID PLUS INT
so the code here recognises the floating point literal and reconsttructs. As an additional benefit 'e' and 'E' are not keywords.
Field Summary |
protected java.util.LinkedList<org.antlr.runtime.Token> |
queue
|
protected java.util.Map<java.lang.Integer,java.lang.String> |
tokenDefMap
|
protected org.antlr.runtime.TokenSource |
tokenSource
|
Constructor Summary |
RetokenizingTokenSource(org.antlr.runtime.TokenSource tokenSource,
java.util.Map<java.lang.Integer,java.lang.String> tokenDefMap)
|
Method Summary |
java.lang.String |
getSourceName()
|
protected boolean |
isInteger(java.lang.String text)
|
org.antlr.runtime.Token |
nextToken()
|
protected org.antlr.runtime.Token |
peek(int index)
Return the token index positions ahead of the next token. |
Methods inherited from class java.lang.Object |
clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait |
tokenSource
protected final org.antlr.runtime.TokenSource tokenSource
tokenDefMap
protected final java.util.Map<java.lang.Integer,java.lang.String> tokenDefMap
queue
protected final java.util.LinkedList<org.antlr.runtime.Token> queue
RetokenizingTokenSource
public RetokenizingTokenSource(org.antlr.runtime.TokenSource tokenSource,
java.util.Map<java.lang.Integer,java.lang.String> tokenDefMap)
getSourceName
public java.lang.String getSourceName()
- Specified by:
getSourceName
in interface org.antlr.runtime.TokenSource
isInteger
protected boolean isInteger(java.lang.String text)
nextToken
public org.antlr.runtime.Token nextToken()
- Specified by:
nextToken
in interface org.antlr.runtime.TokenSource
peek
protected org.antlr.runtime.Token peek(int index)
- Return the token index positions ahead of the next token.
Returns EOF_TOKEN if no such token or if a hidden token interleaves.