module ANTLR3::TokenStream
TokenStream
further extends the abstract-ish base mixin Stream
to add methods specific to navigating token sequences. Thus, it serves as an imitation of the Java interface for token-based streams, which are used by many different components in ANTLR, including parsers and tree parsers.
Token
Streams¶ ↑
Token
streams wrap a sequence of token objects produced by some token source, usually a lexer. They provide the operations required by higher-level recognizers, such as parsers and tree parsers for navigating through the sequence of tokens. Unlike simple character-based streams, such as StringStream
, token-based streams have an additional level of complexity because they must manage the task of “tuning” to a specific token channel.
One of the main advantages of ANTLR-based recognition is the token channel feature, which allows you to hold on to all tokens of interest while only presenting a specific set of interesting tokens to a parser. For example, if you need to hide whitespace and comments from a parser, but hang on to them for some other purpose, you have the lexer assign the comments and whitespace to channel value HIDDEN as it creates the tokens.
When you create a token stream, you can tune it to some specific channel value. Then, all peek
, look
, and consume
operations only yield tokens that have the same value for channel
. The stream skips over any non-matching tokens in between.
The TokenStream
Interface¶ ↑
In addition to the abstract methods and attribute methods provided by the base Stream
module, TokenStream
adds a number of additional method implementation requirements and attributes.
Attributes
the integer channel value to which the stream is “tuned”
expected to return the value of the last marker produced by a call to stream.mark
expected to return the integer index of the stream cursor
expected to return the token source object (such as a lexer) from which all tokens in the stream were retreived
Public Instance Methods
return the stream symbol at index i
# File lib/antlr3/streams.rb, line 337 abstract :at
should take the tokens between start and stop in the sequence, extract their text and return the concatenation of all the text chunks
# File lib/antlr3/streams.rb, line 332 abstract :to_s