class RubyTokenParser

# ↑

RubyTokenParser ↑

Parse Strings containing ruby literals.

@examples

RubyTokenParser.parse("nil")  # => nil
RubyTokenParser.parse(":foo") # => :foo
RubyTokenParser.parse("123")  # => 123
RubyTokenParser.parse("1.5")  # => 1.5
RubyTokenParser.parse("1.5", use_big_decimal: true) # => #<BigDecimal:…,'0.15E1',18(18)>
RubyTokenParser.parse("[1, 2, 3]") # => [1, 2, 3]
RubyTokenParser.parse("{:a => 1, :b => 2}") # => {:a => 1, :b => 2}
RubyTokenParser.parse("(1..3)")

RubyTokenParser recognizes constants and the following literals:

nil                    # nil
true                   # true
false                  # false
-123                   # Fixnum/Bignum (decimal)
0b1011                 # Fixnum/Bignum (binary)
0755                   # Fixnum/Bignum (octal)
0xff                   # Fixnum/Bignum (hexadecimal)
120.30                 # Float (optional: BigDecimal)
1e0                    # Float
"foo"                  # String, no interpolation, but \t etc. work
'foo'                  # String, only \\ and \' are escaped
/foo/                  # Regexp
:foo                   # Symbol
:"foo"                 # Symbol
2012-05-20             # Date
2012-05-20T18:29:52    # DateTime
[Any, Literals, Here]  # Array
{Any => Literals}      # Hash
(1..20)                # Range

@note Limitations

* RubyTokenParser does not support ruby 1.9's `{key: value}` syntax.
* RubyTokenParser does not currently support all of rubys escape
  sequences in strings and symbols.
* Trailing commas in Array and Hash are not supported.

@note BigDecimals

You can instruct RubyTokenParser to parse "12.5" as a bigdecimal
and use "12.5e" to have it parsed as float (short for "12.5e0",
equivalent to "1.25e1")

@note Date & Time

RubyTokenParser supports a subset of ISO-8601 for Date and Time which
are not actual valid ruby literals. The form YYYY-MM-DD (e.g. 2012-05-20)
is translated to a Date object, and YYYY-MM-DD"T"HH:MM:SS (e.g.
2012-05-20T18:29:52) is translated to a Time object.
# ↑

Public Class Methods

new(string, options = nil) click to toggle source
# ↑

initialize ↑

Parse a String, returning the object which it contains.

@param [String] string

The string which should be parsed

@param [nil, Hash] options

An options-hash

@option options [Boolean] :use_big_decimal

Whether to use BigDecimal instead of Float for objects like "1.23".
Defaults to false.

@option options [Boolean] :constant_base

Determines from what constant other constants are searched.

Defaults to Object (nil is treated as Object too, Object
is the toplevel-namespace).
# ↑
# File lib/ruby_token_parser/ruby_token_parser.rb, line 122
def initialize(string, options = nil)
  @string          = string
  options          = options ? options.dup : {}
  @constant_base   = options[:constant_base] # nil means toplevel
  @use_big_decimal = options.delete(:use_big_decimal) { false }
  @scanner         = StringScanner.new(string)
end
parse( string, options = nil, do_raise_an_exception = RubyTokenParser.raise_exception? ) click to toggle source
# ↑

RubyTokenParser.parse ↑

The RubyTokenParser.parse() method will parse a String, and return the (ruby) object which it contains.

@usage example:

RubyTokenParser.parse(":foo") # => :foo

@param [String] string

The (input) String which should be parsed.

@param [nil, Hash] options

An options-hash

@param [Boolean] do_raise_an_exception

This boolean will determine whether we will raise an exception or whether we will not.

@option options [Boolean] :use_big_decimal

Whether to use BigDecimal instead of Float for objects like "1.23".
Defaults to false.

@option options [Boolean] :constant_base

Determines from what constant other constants are searched.
Defaults to Object (nil is treated as Object too, Object
is the toplevel-namespace).

@return [Object] The object in the string.

@raise [RubyTokenParser::SyntaxError]

If the String does not contain exactly one valid literal,
a SyntaxError is raised.

Usage example:

x = RubyTokenParser.parse("[1,2,3]")
# ↑
# File lib/ruby_token_parser/parse.rb, line 47
def self.parse(
    string,
    options = nil,
    do_raise_an_exception = RubyTokenParser.raise_exception?
  )
  # ======================================================================= #
  # === Instantiate a new parser
  # ======================================================================= #
  parser  = new(string, options)
  begin
    value = parser.scan_value
  rescue RubyTokenParser::SyntaxError # Must rescue things such as: @foo = foo
    value = RubyTokenParser::SyntaxError
  end
  if do_raise_an_exception
    unless parser.end_of_string? or
           value.nil?
      # =================================================================== #
      # Raise the Syntax Error.
      # =================================================================== #
      raise SyntaxError,
            "Unexpected superfluous data: #{parser.rest.inspect}"
    end unless value.is_a? Range # Make an exception for Range objects.
  end
  value
end
raise_exception?() click to toggle source
# ↑

RubyTokenParser.raise_exception? ↑

# ↑
# File lib/ruby_token_parser/ruby_token_parser.rb, line 89
def self.raise_exception?
  @do_raise_exception
end
set_do_raise_exception(i = true) click to toggle source
# ↑

RubyTokenParser.set_do_raise_exception ↑

# ↑
# File lib/ruby_token_parser/ruby_token_parser.rb, line 82
def self.set_do_raise_exception(i = true)
  @do_raise_exception = i
end

Public Instance Methods

constant_base()
Alias for: constant_base?
constant_base?() click to toggle source
# ↑

constant_base? ↑

@return [Module, nil]

Where to lookup constants. Nil is toplevel (equivalent to Object).
# ↑
# File lib/ruby_token_parser/ruby_token_parser.rb, line 209
def constant_base?
  @constant_base
end
Also aliased as: constant_base
content?() click to toggle source
# ↑

content? ↑

Reader method over the current value of the scanner.

# ↑
# File lib/ruby_token_parser/ruby_token_parser.rb, line 179
def content?
  @scanner.string
end
end_of_string?() click to toggle source
# ↑

end_of_string? ↑

@return [Boolean]

Whether the scanner reached the end of the string.

# ↑
# File lib/ruby_token_parser/ruby_token_parser.rb, line 161
def end_of_string?
  @scanner.eos?
end
inspect?() click to toggle source
# ↑

inspect? ↑

# ↑
# File lib/ruby_token_parser/ruby_token_parser.rb, line 198
def inspect?
  @scanner.rest.inspect
end
position()
Alias for: position?
position=(i) click to toggle source
# ↑

position= ↑

Moves the scanners position to the given character-index.

@param [Integer] value

The new position of the scanner
# ↑
# File lib/ruby_token_parser/ruby_token_parser.rb, line 150
def position=(i)
  @scanner.pos = i
end
position?() click to toggle source
# ↑

position? ↑

@return [Integer]

The position of the scanner in the string.

# ↑
# File lib/ruby_token_parser/ruby_token_parser.rb, line 137
def position?
  @scanner.pos
end
Also aliased as: position
rest() click to toggle source
# ↑

rest ↑

@return [String] The currently unprocessed rest of the string.

# ↑
# File lib/ruby_token_parser/ruby_token_parser.rb, line 170
def rest
  @scanner.rest
end
scan_value() click to toggle source
# ↑

scan_value ↑

Scans the string for a single value and advances the parsers position.

@return [Object] the scanned value

@raise [RubyTokenParser::SyntaxError]

When no valid ruby object could be scanned at the given position,
a RubyTokenParser::SyntaxError is raised. Alternative you can
disable raising an error by calling:

  RubyTokenParser.set_do_raise_exception(false)
# ↑
# File lib/ruby_token_parser/ruby_token_parser.rb, line 229
def scan_value
  case
  # ======================================================================= #
  # === Handle Ranges                               (range tag, ranges tag)
  # ======================================================================= #
  when (!content?.scan(RRange).empty?)
    _ = content?.delete('(').delete(')').squeeze('.').split('.').map(&:to_i)
    min = _.first
    max = _.last
    Range.new(min, max)
  # ======================================================================= #
  # === Handle Arrays                               (arrays tag, array tag)
  # ======================================================================= #
  when @scanner.scan(RArrayBegin)
    value = []
    @scanner.scan(RArrayVoid)
    if @scanner.scan(RArrayEnd)
      value
    else
      value << scan_value
      while @scanner.scan(RArraySeparator)
        value << scan_value
      end
      unless @scanner.scan(RArrayVoid) && @scanner.scan(RArrayEnd)
        raise SyntaxError, 'Expected ]'
      end
      value
    end
  # ======================================================================= #
  # === Handle Hashes
  #
  # This is quite complicated. We have to scan whether we may find
  # the {} syntax or the end of a hash.
  # ======================================================================= #
  when @scanner.scan(RHashBegin)
    value = {}
    @scanner.scan(RHashVoid)
    if @scanner.scan(RHashEnd)
      value
    else
      if @scanner.scan(RHashKeySymbol)
        key = @scanner[1].to_sym
        @scanner.scan(RHashVoid)
      else
        key = scan_value
        unless @scanner.scan(RHashArrow)
          raise SyntaxError, 'Expected =>'
        end
      end
      val = scan_value
      value[key] = val
      while @scanner.scan(RHashSeparator)
        if @scanner.scan(RHashKeySymbol)
          key = @scanner[1].to_sym
          @scanner.scan(RHashVoid)
        else
          key = scan_value
          raise SyntaxError, 'Expected =>' unless @scanner.scan(RHashArrow)
        end
        val = scan_value
        value[key] = val
      end
      unless @scanner.scan(RHashVoid) && @scanner.scan(RHashEnd)
        raise SyntaxError, 'Expected }'
      end
      value
    end
  # ======================================================================= #
  # === Handle Constants
  #
  # eval() is evil but it may be sane due to the regex, also
  # it's less annoying than deep_const_get.
  #
  # @constant_base can be set via the Hash options[:constant_base].
  # ======================================================================= #
  when @scanner.scan(RConstant)
    eval("#{@constant_base}::#{@scanner.first}")
  # ======================================================================= #
  # === Handle Nil values
  # ======================================================================= #
  when @scanner.scan(RNil)
    nil
  # ======================================================================= #
  # === Handle True values
  # ======================================================================= #
  when @scanner.scan(RTrue) # true tag
    true
  # ======================================================================= #
  # === Handle False values
  # ======================================================================= #
  when @scanner.scan(RFalse) # false tag
    false
  # ======================================================================= #
  # === Handle DateTime values
  # ======================================================================= #
  when @scanner.scan(RDateTime)
    Time.mktime( # Tap into the regex pattern next.
      @scanner[1], @scanner[2],
      @scanner[3], @scanner[4],
      @scanner[5], @scanner[6]
    )
  # ======================================================================= #
  # === Handle Date values
  # ======================================================================= #
  when @scanner.scan(RDate)
    date = @scanner[1].to_i, @scanner[2].to_i, @scanner[3].to_i
    Date.civil(*date)
  # ======================================================================= #
  # === Handle RTime values
  # ======================================================================= #
  when @scanner.scan(RTime)
    now = Time.now
    Time.mktime(
      now.year, now.month, now.day,
      @scanner[1].to_i, @scanner[2].to_i, @scanner[3].to_i
    )
  # ======================================================================= #
  # === Handle Float values
  # ======================================================================= #
  when @scanner.scan(RFloat)
    Float(@scanner.matched.delete('^0-9.e-'))
  # ======================================================================= #
  # === Handle BigDecimal values
  # ======================================================================= #
  when @scanner.scan(RBigDecimal)
    data = @scanner.matched.delete('^0-9.-')
    @use_big_decimal ? BigDecimal(data) : Float(data)
  # ======================================================================= #
  # === Handle OctalInteger values
  # ======================================================================= #
  when @scanner.scan(ROctalInteger)
    # ===================================================================== #
    # We can make use of Integer to turn them into valid ruby objects.
    # ===================================================================== #
    Integer(@scanner.matched.delete('^0-9-'))
  # ======================================================================= #
  # === Handle HexInteger values
  # ======================================================================= #
  when @scanner.scan(RHexInteger)
    Integer(@scanner.matched.delete('^xX0-9A-Fa-f-'))
  # ======================================================================= #
  # === Handle BinaryInteger values
  # ======================================================================= #
  when @scanner.scan(RBinaryInteger)
    Integer(@scanner.matched.delete('^bB01-'))
  # ======================================================================= #
  # === Handle Integer values
  # ======================================================================= #
  when @scanner.scan(RInteger)
    @scanner.matched.delete('^0-9-').to_i
  # ======================================================================= #
  # === Handle Regexp values
  # ======================================================================= #
  when @scanner.scan(RRegexp)
    source = @scanner[1]
    flags  = 0
    lang   = nil
    if @scanner[2]
      flags |= Regexp::IGNORECASE if @scanner[2].include?('i') # Value of 1
      flags |= Regexp::EXTENDED   if @scanner[2].include?('m') # Value of 2
      flags |= Regexp::MULTILINE  if @scanner[2].include?('x') # Value of true
      lang   = @scanner[2].delete('^nNeEsSuU')[-1,1]
    end
    Regexp.new(source, flags, lang)
  # ======================================================================= #
  # === Handle double-quoted string values
  # ======================================================================= #
  when @scanner.scan(RDString)
    @scanner.matched[1..-2].gsub(/\\(?:[0-3]?\d\d?|x[A-Fa-f\d]{2}|.)/) { |m|
      DStringEscapes[m]
    }
  # ======================================================================= #
  # === Handle Symbol values                      (symbol tag, symbols tag)
  # ======================================================================= #
  when @scanner.scan(RSymbol)
    # ===================================================================== #
    # Next, check the first character matched.
    # ===================================================================== #
    case @scanner.matched[1,1] # Might be "f".
    # ===================================================================== #
    # If it is a '"' quote, enter here.
    # ===================================================================== #
    when '"'
      @scanner.matched[2..-2].gsub(/\\(?:[0-3]?\d\d?|x[A-Fa-f\d]{2}|.)/) { |m|
        DStringEscapes[m]
      }.to_sym
    # ===================================================================== #
    # If it is a "'" quote, enter here.
    # ===================================================================== #
    when "'"
      @scanner.matched[2..-2].gsub(/\\'/, "'").gsub(/\\\\/, "\\").to_sym
    else # Default here. Match all but the leading ':'
      @scanner.matched[1..-1].to_sym
    end
  # ======================================================================= #
  # === Handle single-quoted string values
  # ======================================================================= #
  when @scanner.scan(RSString)
    @scanner.matched[1..-2].gsub(/\\'/, "'").gsub(/\\\\/, "\\")
  # ======================================================================= #
  # === Handle everything else
  #
  # This can lead to a runtime error, so we must raise a SyntaxError.
  # ======================================================================= #
  else # else tag
    raise SyntaxError, "Unrecognized pattern: #{inspect?}"
  end
end
use_big_decimal()
Alias for: use_big_decimal?
use_big_decimal?() click to toggle source
# ↑

use_big_decimal? ↑

@return [Boolean]

True if "1.25" should be parsed into a big-decimal,
false if it should be parsed as Float.
# ↑
# File lib/ruby_token_parser/ruby_token_parser.rb, line 191
def use_big_decimal?
  @use_big_decimal
end
Also aliased as: use_big_decimal