class RubyTokenParser
#¶ ↑
RubyTokenParser
¶ ↑
Parse Strings containing ruby literals.
@examples
RubyTokenParser.parse("nil") # => nil RubyTokenParser.parse(":foo") # => :foo RubyTokenParser.parse("123") # => 123 RubyTokenParser.parse("1.5") # => 1.5 RubyTokenParser.parse("1.5", use_big_decimal: true) # => #<BigDecimal:…,'0.15E1',18(18)> RubyTokenParser.parse("[1, 2, 3]") # => [1, 2, 3] RubyTokenParser.parse("{:a => 1, :b => 2}") # => {:a => 1, :b => 2} RubyTokenParser.parse("(1..3)")
RubyTokenParser
recognizes constants and the following literals:
nil # nil true # true false # false -123 # Fixnum/Bignum (decimal) 0b1011 # Fixnum/Bignum (binary) 0755 # Fixnum/Bignum (octal) 0xff # Fixnum/Bignum (hexadecimal) 120.30 # Float (optional: BigDecimal) 1e0 # Float "foo" # String, no interpolation, but \t etc. work 'foo' # String, only \\ and \' are escaped /foo/ # Regexp :foo # Symbol :"foo" # Symbol 2012-05-20 # Date 2012-05-20T18:29:52 # DateTime [Any, Literals, Here] # Array {Any => Literals} # Hash (1..20) # Range
@note Limitations
* RubyTokenParser does not support ruby 1.9's `{key: value}` syntax. * RubyTokenParser does not currently support all of rubys escape sequences in strings and symbols. * Trailing commas in Array and Hash are not supported.
@note BigDecimals
You can instruct RubyTokenParser to parse "12.5" as a bigdecimal and use "12.5e" to have it parsed as float (short for "12.5e0", equivalent to "1.25e1")
@note Date & Time
RubyTokenParser supports a subset of ISO-8601 for Date and Time which are not actual valid ruby literals. The form YYYY-MM-DD (e.g. 2012-05-20) is translated to a Date object, and YYYY-MM-DD"T"HH:MM:SS (e.g. 2012-05-20T18:29:52) is translated to a Time object.
#¶ ↑
Public Class Methods
#¶ ↑
initialize¶ ↑
Parse a String, returning the object which it contains.
@param [String] string
The string which should be parsed
@param [nil, Hash] options
An options-hash
@option options [Boolean] :use_big_decimal
Whether to use BigDecimal instead of Float for objects like "1.23". Defaults to false.
@option options [Boolean] :constant_base
Determines from what constant other constants are searched. Defaults to Object (nil is treated as Object too, Object is the toplevel-namespace).
#¶ ↑
# File lib/ruby_token_parser/ruby_token_parser.rb, line 122 def initialize(string, options = nil) @string = string options = options ? options.dup : {} @constant_base = options[:constant_base] # nil means toplevel @use_big_decimal = options.delete(:use_big_decimal) { false } @scanner = StringScanner.new(string) end
#¶ ↑
RubyTokenParser.parse
¶ ↑
The RubyTokenParser.parse()
method will parse a String, and return the (ruby) object which it contains.
@usage example:
RubyTokenParser.parse(":foo") # => :foo
@param [String] string
The (input) String which should be parsed.
@param [nil, Hash] options
An options-hash
@param [Boolean] do_raise_an_exception
This boolean will determine whether we will raise an exception or whether we will not.
@option options [Boolean] :use_big_decimal
Whether to use BigDecimal instead of Float for objects like "1.23". Defaults to false.
@option options [Boolean] :constant_base
Determines from what constant other constants are searched. Defaults to Object (nil is treated as Object too, Object is the toplevel-namespace).
@return [Object] The object in the string.
@raise [RubyTokenParser::SyntaxError]
If the String does not contain exactly one valid literal, a SyntaxError is raised.
Usage example:
x = RubyTokenParser.parse("[1,2,3]")
#¶ ↑
# File lib/ruby_token_parser/parse.rb, line 47 def self.parse( string, options = nil, do_raise_an_exception = RubyTokenParser.raise_exception? ) # ======================================================================= # # === Instantiate a new parser # ======================================================================= # parser = new(string, options) begin value = parser.scan_value rescue RubyTokenParser::SyntaxError # Must rescue things such as: @foo = foo value = RubyTokenParser::SyntaxError end if do_raise_an_exception unless parser.end_of_string? or value.nil? # =================================================================== # # Raise the Syntax Error. # =================================================================== # raise SyntaxError, "Unexpected superfluous data: #{parser.rest.inspect}" end unless value.is_a? Range # Make an exception for Range objects. end value end
Public Instance Methods
#¶ ↑
constant_base
?¶ ↑
@return [Module, nil]
Where to lookup constants. Nil is toplevel (equivalent to Object).
#¶ ↑
# File lib/ruby_token_parser/ruby_token_parser.rb, line 209 def constant_base? @constant_base end
#¶ ↑
position?¶ ↑
@return [Integer]
The position of the scanner in the string.
#¶ ↑
# File lib/ruby_token_parser/ruby_token_parser.rb, line 137 def position? @scanner.pos end
#¶ ↑
scan_value
¶ ↑
Scans the string for a single value and advances the parsers position.
@return [Object] the scanned value
@raise [RubyTokenParser::SyntaxError]
When no valid ruby object could be scanned at the given position, a RubyTokenParser::SyntaxError is raised. Alternative you can disable raising an error by calling: RubyTokenParser.set_do_raise_exception(false)
#¶ ↑
# File lib/ruby_token_parser/ruby_token_parser.rb, line 229 def scan_value case # ======================================================================= # # === Handle Ranges (range tag, ranges tag) # ======================================================================= # when (!content?.scan(RRange).empty?) _ = content?.delete('(').delete(')').squeeze('.').split('.').map(&:to_i) min = _.first max = _.last Range.new(min, max) # ======================================================================= # # === Handle Arrays (arrays tag, array tag) # ======================================================================= # when @scanner.scan(RArrayBegin) value = [] @scanner.scan(RArrayVoid) if @scanner.scan(RArrayEnd) value else value << scan_value while @scanner.scan(RArraySeparator) value << scan_value end unless @scanner.scan(RArrayVoid) && @scanner.scan(RArrayEnd) raise SyntaxError, 'Expected ]' end value end # ======================================================================= # # === Handle Hashes # # This is quite complicated. We have to scan whether we may find # the {} syntax or the end of a hash. # ======================================================================= # when @scanner.scan(RHashBegin) value = {} @scanner.scan(RHashVoid) if @scanner.scan(RHashEnd) value else if @scanner.scan(RHashKeySymbol) key = @scanner[1].to_sym @scanner.scan(RHashVoid) else key = scan_value unless @scanner.scan(RHashArrow) raise SyntaxError, 'Expected =>' end end val = scan_value value[key] = val while @scanner.scan(RHashSeparator) if @scanner.scan(RHashKeySymbol) key = @scanner[1].to_sym @scanner.scan(RHashVoid) else key = scan_value raise SyntaxError, 'Expected =>' unless @scanner.scan(RHashArrow) end val = scan_value value[key] = val end unless @scanner.scan(RHashVoid) && @scanner.scan(RHashEnd) raise SyntaxError, 'Expected }' end value end # ======================================================================= # # === Handle Constants # # eval() is evil but it may be sane due to the regex, also # it's less annoying than deep_const_get. # # @constant_base can be set via the Hash options[:constant_base]. # ======================================================================= # when @scanner.scan(RConstant) eval("#{@constant_base}::#{@scanner.first}") # ======================================================================= # # === Handle Nil values # ======================================================================= # when @scanner.scan(RNil) nil # ======================================================================= # # === Handle True values # ======================================================================= # when @scanner.scan(RTrue) # true tag true # ======================================================================= # # === Handle False values # ======================================================================= # when @scanner.scan(RFalse) # false tag false # ======================================================================= # # === Handle DateTime values # ======================================================================= # when @scanner.scan(RDateTime) Time.mktime( # Tap into the regex pattern next. @scanner[1], @scanner[2], @scanner[3], @scanner[4], @scanner[5], @scanner[6] ) # ======================================================================= # # === Handle Date values # ======================================================================= # when @scanner.scan(RDate) date = @scanner[1].to_i, @scanner[2].to_i, @scanner[3].to_i Date.civil(*date) # ======================================================================= # # === Handle RTime values # ======================================================================= # when @scanner.scan(RTime) now = Time.now Time.mktime( now.year, now.month, now.day, @scanner[1].to_i, @scanner[2].to_i, @scanner[3].to_i ) # ======================================================================= # # === Handle Float values # ======================================================================= # when @scanner.scan(RFloat) Float(@scanner.matched.delete('^0-9.e-')) # ======================================================================= # # === Handle BigDecimal values # ======================================================================= # when @scanner.scan(RBigDecimal) data = @scanner.matched.delete('^0-9.-') @use_big_decimal ? BigDecimal(data) : Float(data) # ======================================================================= # # === Handle OctalInteger values # ======================================================================= # when @scanner.scan(ROctalInteger) # ===================================================================== # # We can make use of Integer to turn them into valid ruby objects. # ===================================================================== # Integer(@scanner.matched.delete('^0-9-')) # ======================================================================= # # === Handle HexInteger values # ======================================================================= # when @scanner.scan(RHexInteger) Integer(@scanner.matched.delete('^xX0-9A-Fa-f-')) # ======================================================================= # # === Handle BinaryInteger values # ======================================================================= # when @scanner.scan(RBinaryInteger) Integer(@scanner.matched.delete('^bB01-')) # ======================================================================= # # === Handle Integer values # ======================================================================= # when @scanner.scan(RInteger) @scanner.matched.delete('^0-9-').to_i # ======================================================================= # # === Handle Regexp values # ======================================================================= # when @scanner.scan(RRegexp) source = @scanner[1] flags = 0 lang = nil if @scanner[2] flags |= Regexp::IGNORECASE if @scanner[2].include?('i') # Value of 1 flags |= Regexp::EXTENDED if @scanner[2].include?('m') # Value of 2 flags |= Regexp::MULTILINE if @scanner[2].include?('x') # Value of true lang = @scanner[2].delete('^nNeEsSuU')[-1,1] end Regexp.new(source, flags, lang) # ======================================================================= # # === Handle double-quoted string values # ======================================================================= # when @scanner.scan(RDString) @scanner.matched[1..-2].gsub(/\\(?:[0-3]?\d\d?|x[A-Fa-f\d]{2}|.)/) { |m| DStringEscapes[m] } # ======================================================================= # # === Handle Symbol values (symbol tag, symbols tag) # ======================================================================= # when @scanner.scan(RSymbol) # ===================================================================== # # Next, check the first character matched. # ===================================================================== # case @scanner.matched[1,1] # Might be "f". # ===================================================================== # # If it is a '"' quote, enter here. # ===================================================================== # when '"' @scanner.matched[2..-2].gsub(/\\(?:[0-3]?\d\d?|x[A-Fa-f\d]{2}|.)/) { |m| DStringEscapes[m] }.to_sym # ===================================================================== # # If it is a "'" quote, enter here. # ===================================================================== # when "'" @scanner.matched[2..-2].gsub(/\\'/, "'").gsub(/\\\\/, "\\").to_sym else # Default here. Match all but the leading ':' @scanner.matched[1..-1].to_sym end # ======================================================================= # # === Handle single-quoted string values # ======================================================================= # when @scanner.scan(RSString) @scanner.matched[1..-2].gsub(/\\'/, "'").gsub(/\\\\/, "\\") # ======================================================================= # # === Handle everything else # # This can lead to a runtime error, so we must raise a SyntaxError. # ======================================================================= # else # else tag raise SyntaxError, "Unrecognized pattern: #{inspect?}" end end
#¶ ↑
use_big_decimal
?¶ ↑
@return [Boolean]
True if "1.25" should be parsed into a big-decimal, false if it should be parsed as Float.
#¶ ↑
# File lib/ruby_token_parser/ruby_token_parser.rb, line 191 def use_big_decimal? @use_big_decimal end