public class RetokenizingTokenSource
extends Object
RetokenizingTokenSource accomodates backtracking limitations in the ANTRL lexer when used from Xtext.
The problem is the three overlapping syntaxes
INT.INT leading to a Floating Point lteral
INT..INT leading to a Collection range
INT.ID leading to a numeric navigation
ANTLR proceeds to INT. but won't backup when the character after the . is bad.
The code here allows the basic lexer to be ignorant of floating point syntax so that it correctly parses
INT.INT as INT DOT INT
INT..INT as INT DOTDOT INT
INT.INTe+INT as INT DOT INT ID PLUS INT
so the code here recognises the floating point literal and reconsttructs. As an additional benefit 'e' and 'E' are not keywords.