class Pinpoint::Format::Tokenizer
Attributes
Protected: Reader for tokenable
Public Class Methods
Public: Initializes a Tokenizer
with a given String
# File lib/pinpoint/format/tokenizer.rb, line 17 def initialize(string_with_tokens) self.tokenable = string_with_tokens end
Public Instance Methods
Public: From the beginning of the tokenable String, it reads each token and passes it to the block.
Yields each successive Token
to the given block
Raises subclasses of Pinpoint::Format::ParseError
upon various syntax
errors
Returns nothing
# File lib/pinpoint/format/tokenizer.rb, line 32 def each tokenable.reset while current_token = next_token yield current_token end end
Protected Instance Methods
Protected: Reads the token_map
for the Tokenizer
Returns the token_map
if it is set, otherwise sets the token_map
to the default and returns it.
# File lib/pinpoint/format/tokenizer.rb, line 83 def token_map @token_map ||= { literal: /[^\(\)%]+/, name: /%n/, street: /%s/, locality: /%l/, province: /%p/, postal_code: /%z/, country: /%c/, percent: /%%/, left_paren: /%\(/, right_paren: /%\)/, group_start: /\(/, group_end: /\)/, } end
Protected: Sets tokenable but first checks to see if it needs to be wrapped in an IO object and does so if necessary.
# File lib/pinpoint/format/tokenizer.rb, line 58 def tokenable=(value) io = wrap_in_io_if_needed(value) @tokenable = StringScanner.new(io.read) end
Private Instance Methods
Private: Can locate the next token in the tokenable String
Example
# When a Token can be found next_token # => <Token type: :token_type, value: 'my_token'> # When a Token cannot be found next_token # => Pinpoint::Format::ParseError # When there is nothing left to process next_token # => false
Returns a Token
if a match is found Returns FalseClass if there is nothing left to process Raises Pinpoint::Format::ParseError
if an unknown Token
is encountered
# File lib/pinpoint/format/tokenizer.rb, line 150 def next_token return if tokenable.eos? case when text = tokenable.scan(token_map[:literal]) then Token.new(:literal, text) when text = tokenable.scan(token_map[:name]) then Token.new(:name, text) when text = tokenable.scan(token_map[:street]) then Token.new(:street, text) when text = tokenable.scan(token_map[:locality]) then Token.new(:locality, text) when text = tokenable.scan(token_map[:province]) then Token.new(:province, text) when text = tokenable.scan(token_map[:postal_code]) then Token.new(:postal_code, text) when text = tokenable.scan(token_map[:country]) then Token.new(:country, text) when text = tokenable.scan(token_map[:percent]) then Token.new(:literal, '%') when text = tokenable.scan(token_map[:left_paren]) then Token.new(:literal, '(') when text = tokenable.scan(token_map[:right_paren]) then Token.new(:literal, ')') when text = tokenable.scan(token_map[:group_start]) then Token.new(:group_start, text) when text = tokenable.scan(token_map[:group_end]) then Token.new(:group_end, text) else fail ParseError, "Cannot parse the remainder of the tokenable string: '#{tokenable.rest}'" end end
Private: Checks to see if the value passed in is an IO or something else. If it’s an IO, it remains unmodified, otherwise the value is converted to a StringIO.
Example
wrap_in_io_if_needed(my_io_object) # => my_io_object wrap_in_io_if_needed('my_string') # => <StringIO contents: 'my_string'> wrap_in_io_if_needed(1) # => <StringIO contents: '1'>
Returns an IO object.
# File lib/pinpoint/format/tokenizer.rb, line 120 def wrap_in_io_if_needed(value) case value when IO value else StringIO.new value.to_s end end