org.eclipse.ocl.examples.xtext.essentialocl.services
Class RetokenizingTokenSource

java.lang.Object
  org.eclipse.ocl.examples.xtext.essentialocl.services.RetokenizingTokenSource

All Implemented Interfaces:

org.antlr.runtime.TokenSource

public class RetokenizingTokenSource
extends java.lang.Object
implements org.antlr.runtime.TokenSource

RetokenizingTokenSource accomodates backtracking limitations in the ANTRL lexer when used from Xtext. The problem is the three overlapping syntaxes INT.INT leading to a Floating Point lteral INT..INT leading to a Collection range INT.ID leading to a numeric navigation ANTLR proceeds to INT. but won't backup when the character after the . is bad. The code here allows the basic lexer to be ignorant of floating point syntax so that it correctly parses INT.INT as INT DOT INT INT..INT as INT DOTDOT INT INT.INTe+INT as INT DOT INT ID PLUS INT so the code here recognises the floating point literal and reconsttructs. As an additional benefit 'e' and 'E' are not keywords.

Field Summary

protected java.util.LinkedList<org.antlr.runtime.Token>	queue
protected java.util.Map<java.lang.Integer,java.lang.String>	tokenDefMap
protected org.antlr.runtime.TokenSource	tokenSource

Constructor Summary

RetokenizingTokenSource(org.antlr.runtime.TokenSource tokenSource, java.util.Map<java.lang.Integer,java.lang.String> tokenDefMap)

Method Summary

java.lang.String	getSourceName()
protected boolean	isInteger(java.lang.String text)
org.antlr.runtime.Token	nextToken()
protected org.antlr.runtime.Token	peek(int index) Return the token index positions ahead of the next token.

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Detail

tokenSource

protected final org.antlr.runtime.TokenSource tokenSource

tokenDefMap

protected final java.util.Map<java.lang.Integer,java.lang.String> tokenDefMap

queue

protected final java.util.LinkedList<org.antlr.runtime.Token> queue

Constructor Detail

RetokenizingTokenSource

public RetokenizingTokenSource(org.antlr.runtime.TokenSource tokenSource,
                               java.util.Map<java.lang.Integer,java.lang.String> tokenDefMap)

Method Detail

getSourceName

public java.lang.String getSourceName()

Specified by:

getSourceName in interface org.antlr.runtime.TokenSource

isInteger

protected boolean isInteger(java.lang.String text)

nextToken

public org.antlr.runtime.Token nextToken()

Specified by:

nextToken in interface org.antlr.runtime.TokenSource

peek

protected org.antlr.runtime.Token peek(int index)

Return the token index positions ahead of the next token. Returns EOF_TOKEN if no such token or if a hidden token interleaves.