class Pinpoint::Format::Tokenizer

Attributes

token_map[W]

Public: Allows the tokens used for the Tokenizer to be overridden.

The value passed must be a Hash with keys representing the title of the token and values containing the Regex which represents a match for that Token.

Example

token_map = { alphanumeric: /[A-Za-z0-9]/
tokenable[R]

Protected: Reader for tokenable

Public Class Methods

new(string_with_tokens) click to toggle source

Public: Initializes a Tokenizer with a given String

# File lib/pinpoint/format/tokenizer.rb, line 17
def initialize(string_with_tokens)
  self.tokenable = string_with_tokens
end

Public Instance Methods

each() { |current_token| ... } click to toggle source

Public: From the beginning of the tokenable String, it reads each token and passes it to the block.

Yields each successive Token to the given block

Raises subclasses of Pinpoint::Format::ParseError upon various syntax

errors

Returns nothing

# File lib/pinpoint/format/tokenizer.rb, line 32
def each
  tokenable.reset

  while current_token = next_token
    yield current_token
  end
end
to_token_list() click to toggle source

Public: Wraps the Array of Tokens in a TokenList

Returns a TokenList

# File lib/pinpoint/format/tokenizer.rb, line 45
def to_token_list
  TokenList.new(to_a)
end

Protected Instance Methods

token_map() click to toggle source

Protected: Reads the token_map for the Tokenizer

Returns the token_map if it is set, otherwise sets the token_map to the default and returns it.

# File lib/pinpoint/format/tokenizer.rb, line 83
def token_map
  @token_map ||= {
    literal:     /[^\(\)%]+/,
    name:        /%n/,
    street:      /%s/,
    locality:    /%l/,
    province:    /%p/,
    postal_code: /%z/,
    country:     /%c/,
    percent:     /%%/,
    left_paren:  /%\(/,
    right_paren: /%\)/,
    group_start: /\(/,
    group_end:   /\)/,
  }
end
tokenable=(value) click to toggle source

Protected: Sets tokenable but first checks to see if it needs to be wrapped in an IO object and does so if necessary.

# File lib/pinpoint/format/tokenizer.rb, line 58
def tokenable=(value)
  io         = wrap_in_io_if_needed(value)

  @tokenable = StringScanner.new(io.read)
end

Private Instance Methods

next_token() click to toggle source

Private: Can locate the next token in the tokenable String

Example

# When a Token can be found
next_token
# => <Token type: :token_type, value: 'my_token'>

# When a Token cannot be found
next_token
# => Pinpoint::Format::ParseError

# When there is nothing left to process
next_token
# => false

Returns a Token if a match is found Returns FalseClass if there is nothing left to process Raises Pinpoint::Format::ParseError if an unknown Token is encountered

# File lib/pinpoint/format/tokenizer.rb, line 150
def next_token
  return if tokenable.eos?

  case
  when text = tokenable.scan(token_map[:literal]) then     Token.new(:literal,     text)
  when text = tokenable.scan(token_map[:name]) then        Token.new(:name,        text)
  when text = tokenable.scan(token_map[:street]) then      Token.new(:street,      text)
  when text = tokenable.scan(token_map[:locality]) then    Token.new(:locality,    text)
  when text = tokenable.scan(token_map[:province]) then    Token.new(:province,    text)
  when text = tokenable.scan(token_map[:postal_code]) then Token.new(:postal_code, text)
  when text = tokenable.scan(token_map[:country]) then     Token.new(:country,     text)
  when text = tokenable.scan(token_map[:percent]) then     Token.new(:literal,     '%')
  when text = tokenable.scan(token_map[:left_paren]) then  Token.new(:literal,     '(')
  when text = tokenable.scan(token_map[:right_paren]) then Token.new(:literal,     ')')
  when text = tokenable.scan(token_map[:group_start]) then Token.new(:group_start, text)
  when text = tokenable.scan(token_map[:group_end]) then   Token.new(:group_end,   text)
  else
    fail ParseError, "Cannot parse the remainder of the tokenable string: '#{tokenable.rest}'"
  end
end
wrap_in_io_if_needed(value) click to toggle source

Private: Checks to see if the value passed in is an IO or something else. If it’s an IO, it remains unmodified, otherwise the value is converted to a StringIO.

Example

wrap_in_io_if_needed(my_io_object)
# => my_io_object

wrap_in_io_if_needed('my_string')
# => <StringIO contents: 'my_string'>

wrap_in_io_if_needed(1)
# => <StringIO contents: '1'>

Returns an IO object.

# File lib/pinpoint/format/tokenizer.rb, line 120
def wrap_in_io_if_needed(value)
  case value
  when IO
    value
  else
    StringIO.new value.to_s
  end
end