class ANTLR3::CommonToken
The base class for the standard implementation of Token
. It is implemented as a simple Struct as tokens are basically simple data structures binding together a bunch of different information and Structs are slightly faster than a standard Object
with accessor methods implementation.
By default, ANTLR generated ruby code will provide a customized subclass of CommonToken
to track token-type names efficiently for debugging, inspection, and general utility. Thus code generated for a standard combo lexer-parser grammar named XYZ will have a base module named XYZ and a customized CommonToken
subclass named XYZ::Token.
Here is the token structure attribute list in order:
-
type
-
channel
-
text
-
input
-
start
-
stop
-
index
-
line
-
column
Constants
- DEFAULT_VALUES
Public Class Methods
# File lib/antlr3/token.rb, line 256 def self.create( fields = {} ) fields = DEFAULT_VALUES.merge( fields ) args = members.map { |name| fields[ name.to_sym ] } new( *args ) end
allows you to make a copy of a token with a different class
# File lib/antlr3/token.rb, line 263 def self.from_token( token ) new( token.type, token.channel, token.text ? token.text.clone : nil, token.input, token.start, token.stop, -1, token.line, token.column ) end
# File lib/antlr3/token.rb, line 270 def initialize( type = nil, channel = DEFAULT_CHANNEL, text = nil, input = nil, start = nil, stop = nil, index = -1, line = 0, column = -1 ) super block_given? and yield( self ) self.text.nil? && self.start && self.stop and self.text = self.input.substring( self.start, self.stop ) end
# File lib/antlr3/token.rb, line 252 def self.token_name( type ) BUILT_IN_TOKEN_NAMES[ type ] end