module ANTLR3::TokenStream

TokenStream further extends the abstract-ish base mixin Stream to add methods specific to navigating token sequences. Thus, it serves as an imitation of the Java interface for token-based streams, which are used by many different components in ANTLR, including parsers and tree parsers.

Token Streams

Token streams wrap a sequence of token objects produced by some token source, usually a lexer. They provide the operations required by higher-level recognizers, such as parsers and tree parsers for navigating through the sequence of tokens. Unlike simple character-based streams, such as StringStream, token-based streams have an additional level of complexity because they must manage the task of “tuning” to a specific token channel.

One of the main advantages of ANTLR-based recognition is the token channel feature, which allows you to hold on to all tokens of interest while only presenting a specific set of interesting tokens to a parser. For example, if you need to hide whitespace and comments from a parser, but hang on to them for some other purpose, you have the lexer assign the comments and whitespace to channel value HIDDEN as it creates the tokens.

When you create a token stream, you can tune it to some specific channel value. Then, all peek, look, and consume operations only yield tokens that have the same value for channel. The stream skips over any non-matching tokens in between.

The TokenStream Interface

In addition to the abstract methods and attribute methods provided by the base Stream module, TokenStream adds a number of additional method implementation requirements and attributes.

Attributes

channel[RW]

the integer channel value to which the stream is “tuned”

last_marker[R]

expected to return the value of the last marker produced by a call to stream.mark

position[R]

expected to return the integer index of the stream cursor

token_source[R]

expected to return the token source object (such as a lexer) from which all tokens in the stream were retreived

Public Instance Methods

at(() click to toggle source

return the stream symbol at index i

# File lib/antlr3/streams.rb, line 337
abstract :at
to_s(start=0,stop=tokens.length-1)() click to toggle source

should take the tokens between start and stop in the sequence, extract their text and return the concatenation of all the text chunks

# File lib/antlr3/streams.rb, line 332
abstract :to_s