class ANTLR3::CommonToken

The base class for the standard implementation of Token. It is implemented as a simple Struct as tokens are basically simple data structures binding together a bunch of different information and Structs are slightly faster than a standard Object with accessor methods implementation.

By default, ANTLR generated ruby code will provide a customized subclass of CommonToken to track token-type names efficiently for debugging, inspection, and general utility. Thus code generated for a standard combo lexer-parser grammar named XYZ will have a base module named XYZ and a customized CommonToken subclass named XYZ::Token.

Here is the token structure attribute list in order:

Constants

DEFAULT_VALUES

Public Class Methods

create( fields = {} ) click to toggle source
# File lib/antlr3/token.rb, line 256
def self.create( fields = {} )
  fields = DEFAULT_VALUES.merge( fields )
  args = members.map { |name| fields[ name.to_sym ] }
  new( *args )
end
from_token( token ) click to toggle source

allows you to make a copy of a token with a different class

# File lib/antlr3/token.rb, line 263
def self.from_token( token )
  new( 
    token.type,  token.channel, token.text ? token.text.clone : nil,
    token.input, token.start,   token.stop, -1, token.line, token.column
  )
end
new( type = nil, channel = DEFAULT_CHANNEL, text = nil, input = nil, start = nil, stop = nil, index = -1, line = 0, column = -1 ) { |self| ... } click to toggle source
Calls superclass method
# File lib/antlr3/token.rb, line 270
def initialize( type = nil, channel = DEFAULT_CHANNEL, text = nil,
               input = nil, start = nil, stop = nil, index = -1,
               line = 0, column = -1 )
  super
  block_given? and yield( self )
  self.text.nil? && self.start && self.stop and
    self.text = self.input.substring( self.start, self.stop )
end
token_name( type ) click to toggle source
# File lib/antlr3/token.rb, line 252
def self.token_name( type )
  BUILT_IN_TOKEN_NAMES[ type ]
end