class Kaitai::Struct::Stream

Kaitai::Struct::Stream is an implementation of {github.com/kaitai-io/kaitai_struct/wiki/Kaitai-Struct-stream-API Kaitai Struct stream API} for Ruby. It's implemented as a wrapper for generic IO objects.

It provides a wide variety of simple methods to read (parse) binary representations of primitive types, such as integer and floating point numbers, byte arrays and strings, and also provides stream positioning / navigation methods with unified cross-language and cross-toolkit semantics.

Typically, end users won't access Kaitai Stream class manually, but would describe a binary structure format using .ksy language and then would use Kaitai Struct compiler to generate source code in desired target language. That code, in turn, would use this class and API to do the actual parsing job.

Constants

SIGN_MASK_16
SIGN_MASK_32
SIGN_MASK_64

Public Class Methods

bytes_strip_right(bytes, pad_byte) click to toggle source
# File lib/kaitai/struct/struct.rb, line 420
def self.bytes_strip_right(bytes, pad_byte)
  new_len = bytes.length
  while new_len > 0 and bytes.getbyte(new_len - 1) == pad_byte
    new_len -= 1
  end

  bytes[0, new_len]
end
bytes_terminate(bytes, term, include_term) click to toggle source
# File lib/kaitai/struct/struct.rb, line 429
def self.bytes_terminate(bytes, term, include_term)
  new_len = 0
  max_len = bytes.length
  while bytes.getbyte(new_len) != term and new_len < max_len
    new_len += 1
  end
  new_len += 1 if include_term and new_len < max_len
  bytes[0, new_len]
end
new(arg) click to toggle source

Constructs new Kaitai Stream object. @param arg [String, IO] if String, it will be used as byte array to read data from;

if IO, if will be used literally as source of data
# File lib/kaitai/struct/struct.rb, line 98
def initialize(arg)
  if arg.is_a?(String)
    @_io = StringIO.new(arg)
  elsif arg.is_a?(IO)
    @_io = arg
  else
    raise TypeError.new('can be initialized with IO or String only')
  end
  align_to_byte
end
open(filename) click to toggle source

Convenience method to create a Kaitai Stream object, opening a local file with a given filename. @param filename [String] local file to open

# File lib/kaitai/struct/struct.rb, line 113
def self.open(filename)
  self.new(File.open(filename, 'rb:ASCII-8BIT'))
end
process_rotate_left(data, amount, group_size) click to toggle source

Performs a circular left rotation shift for a given buffer by a given amount of bits, using groups of groupSize bytes each time. Right circular rotation should be performed using this procedure with corrected amount. @param data [String] source data to process @param amount [Fixnum] number of bits to shift by @param group_size [Fixnum] number of bytes per group to shift @return [String] copy of source array with requested shift applied

# File lib/kaitai/struct/struct.rb, line 496
def self.process_rotate_left(data, amount, group_size)
  raise NotImplementedError.new("unable to rotate group #{group_size} bytes yet") unless group_size == 1

  mask = group_size * 8 - 1
  anti_amount = -amount & mask

  # NB: actually, left bit shift (<<) in Ruby would have required
  # truncation to type_bits size (i.e. something like "& 0xff" for
  # group_size == 8), but we can skip this one, because later these
  # number would be packed with Array#pack, which will do truncation
  # anyway

  data.bytes.map { |x| (x << amount) | (x >> anti_amount) }.pack('C*')
end
process_xor_many(data, key) click to toggle source

Performs a XOR processing with given data, XORing every byte of input with a key array, repeating key array many times, if necessary (i.e. if data array is longer than key array). Uses pure Ruby implementation suggested by [Thomas Leitner](github.com/gettalong), borrowed from github.com/fny/xorcist/blob/master/bin/benchmark @param data [String] data to process @param key [String] array of bytes to XOR with @return [String] processed data

# File lib/kaitai/struct/struct.rb, line 472
def self.process_xor_many(data, key)
  out = data.dup
  kl = key.length
  ki = 0
  i = 0
  max = data.length
  while i < max
    out.setbyte(i, data.getbyte(i) ^ key.getbyte(ki))
    ki += 1
    ki = 0 if ki >= kl
    i += 1
  end
  out
end
process_xor_one(data, key) click to toggle source

Performs a XOR processing with given data, XORing every byte of input with a single given value. Uses pure Ruby implementation suggested by [Thomas Leitner](github.com/gettalong), borrowed from github.com/fny/xorcist/blob/master/bin/benchmark @param data [String] data to process @param key [Fixnum] value to XOR with @return [String] processed data

# File lib/kaitai/struct/struct.rb, line 451
def self.process_xor_one(data, key)
  out = data.dup
  i = 0
  max = data.length
  while i < max
    out.setbyte(i, data.getbyte(i) ^ key)
    i += 1
  end
  out
end
resolve_enum(enum_map, value) click to toggle source

Resolves value using enum: if the value is not found in the map, we'll just use literal value per se.

# File lib/kaitai/struct/struct.rb, line 516
def self.resolve_enum(enum_map, value)
  enum_map[value] || value
end

Private Class Methods

format_hex(arr) click to toggle source
# File lib/kaitai/struct/struct.rb, line 531
def self.format_hex(arr)
  arr.unpack('H*')[0].gsub(/(..)/, '\1 ').chop
end

Public Instance Methods

align_to_byte() click to toggle source

@!group Unaligned bit values

# File lib/kaitai/struct/struct.rb, line 290
def align_to_byte
  @bits_left = 0
  @bits = 0
end
close() click to toggle source

Closes underlying IO object.

# File lib/kaitai/struct/struct.rb, line 119
def close
  @_io.close
end
ensure_fixed_contents(expected) click to toggle source

Unused since Kaitai Struct Compiler v0.9+ - compatibility with older versions.

Reads next len bytes from the stream and ensures that they match expected fixed byte array. If they differ, throws a {UnexpectedDataError} runtime exception. @param expected [String] contents to be expected @return [String] read bytes as byte array, which are guaranteed to

equal to expected

@raise [UnexpectedDataError]

# File lib/kaitai/struct/struct.rb, line 413
def ensure_fixed_contents(expected)
  len = expected.bytesize
  actual = @_io.read(len)
  raise UnexpectedDataError.new(actual, expected) if actual != expected
  actual
end
eof?() click to toggle source

Check if stream pointer is at the end of stream. @return [true, false] true if we are located at the end of the stream

# File lib/kaitai/struct/struct.rb, line 131
def eof?; @_io.eof? and @bits_left == 0; end
pos() click to toggle source

Get current position of a stream pointer. @return [Fixnum] pointer position, number of bytes from the beginning of the stream

# File lib/kaitai/struct/struct.rb, line 141
def pos; @_io.pos; end
read_bits_int(n) click to toggle source

Unused since Kaitai Struct Compiler v0.9+ - compatibility with older versions.

# File lib/kaitai/struct/struct.rb, line 325
def read_bits_int(n)
  read_bits_int_be(n)
end
read_bits_int_be(n) click to toggle source
# File lib/kaitai/struct/struct.rb, line 295
def read_bits_int_be(n)
  bits_needed = n - @bits_left
  if bits_needed > 0
    # 1 bit  => 1 byte
    # 8 bits => 1 byte
    # 9 bits => 2 bytes
    bytes_needed = ((bits_needed - 1) / 8) + 1
    buf = read_bytes(bytes_needed)
    buf.each_byte { |byte|
      @bits <<= 8
      @bits |= byte
      @bits_left += 8
    }
  end

  # raw mask with required number of 1s, starting from lowest bit
  mask = (1 << n) - 1
  # shift @bits to align the highest bits with the mask & derive reading result
  shift_bits = @bits_left - n
  res = (@bits >> shift_bits) & mask
  # clear top bits that we've just read => AND with 1s
  @bits_left -= n
  mask = (1 << @bits_left) - 1
  @bits &= mask

  res
end
read_bits_int_le(n) click to toggle source
# File lib/kaitai/struct/struct.rb, line 329
def read_bits_int_le(n)
  bits_needed = n - @bits_left
  if bits_needed > 0
    # 1 bit  => 1 byte
    # 8 bits => 1 byte
    # 9 bits => 2 bytes
    bytes_needed = ((bits_needed - 1) / 8) + 1
    buf = read_bytes(bytes_needed)
    buf.each_byte { |byte|
      @bits |= (byte << @bits_left)
      @bits_left += 8
    }
  end

  # raw mask with required number of 1s, starting from lowest bit
  mask = (1 << n) - 1
  # derive reading result
  res = @bits & mask
  # remove bottom bits that we've just read by shifting
  @bits >>= n
  @bits_left -= n

  res
end
read_bytes(n) click to toggle source

Reads designated number of bytes from the stream. @param n [Fixnum] number of bytes to read @return [String] read bytes as byte array @raise [EOFError] if there were less bytes than requested

available in the stream
# File lib/kaitai/struct/struct.rb, line 364
def read_bytes(n)
  r = @_io.read(n)
  if r
    rl = r.bytesize
  else
    rl = 0
  end
  raise EOFError.new("attempted to read #{n} bytes, got only #{rl}") if rl < n
  r
end
read_bytes_full() click to toggle source

Reads all the remaining bytes in a stream as byte array. @return [String] all remaining bytes in a stream as byte array

# File lib/kaitai/struct/struct.rb, line 378
def read_bytes_full
  @_io.read
end
read_bytes_term(term, include_term, consume_term, eos_error) click to toggle source
# File lib/kaitai/struct/struct.rb, line 382
def read_bytes_term(term, include_term, consume_term, eos_error)
  r = ''
  loop {
    if @_io.eof?
      if eos_error
        raise EOFError.new("end of stream reached, but no terminator #{term} found")
      else
        return r
      end
    end
    c = @_io.getc
    if c.ord == term
      r << c if include_term
      @_io.seek(@_io.pos - 1) unless consume_term
      return r
    end
    r << c
  }
end
read_f4be() click to toggle source

Big-endian


# File lib/kaitai/struct/struct.rb, line 266
def read_f4be
  read_bytes(4).unpack('g')[0]
end
read_f4le() click to toggle source

Little-endian


# File lib/kaitai/struct/struct.rb, line 278
def read_f4le
  read_bytes(4).unpack('e')[0]
end
read_f8be() click to toggle source
# File lib/kaitai/struct/struct.rb, line 270
def read_f8be
  read_bytes(8).unpack('G')[0]
end
read_f8le() click to toggle source
# File lib/kaitai/struct/struct.rb, line 282
def read_f8le
  read_bytes(8).unpack('E')[0]
end
read_s1() click to toggle source

Signed


# File lib/kaitai/struct/struct.rb, line 156
def read_s1
  read_bytes(1).unpack('c')[0]
end
read_s2be() click to toggle source

.….….….….….….….….….….….….….….….….….… Big-endian .….….….….….….….….….….….….….….….….….…

# File lib/kaitai/struct/struct.rb, line 164
def read_s2be
  to_signed(read_u2be, SIGN_MASK_16)
end
read_s2le() click to toggle source

.….….….….….….….….….….….….….….….….….… Little-endian .….….….….….….….….….….….….….….….….….…

# File lib/kaitai/struct/struct.rb, line 186
def read_s2le
  to_signed(read_u2le, SIGN_MASK_16)
end
read_s4be() click to toggle source
# File lib/kaitai/struct/struct.rb, line 168
def read_s4be
  to_signed(read_u4be, SIGN_MASK_32)
end
read_s4le() click to toggle source
# File lib/kaitai/struct/struct.rb, line 190
def read_s4le
  to_signed(read_u4le, SIGN_MASK_32)
end
read_s8be() click to toggle source
# File lib/kaitai/struct/struct.rb, line 173
def read_s8be
  read_bytes(8).unpack('q')[0]
end
read_s8le() click to toggle source
# File lib/kaitai/struct/struct.rb, line 195
def read_s8le
  read_bytes(8).unpack('q')[0]
end
read_u1() click to toggle source

Unsigned


# File lib/kaitai/struct/struct.rb, line 208
def read_u1
  read_bytes(1).unpack('C')[0]
end
read_u2be() click to toggle source

.….….….….….….….….….….….….….….….….….… Big-endian .….….….….….….….….….….….….….….….….….…

# File lib/kaitai/struct/struct.rb, line 216
def read_u2be
  read_bytes(2).unpack('n')[0]
end
read_u2le() click to toggle source

.….….….….….….….….….….….….….….….….….… Little-endian .….….….….….….….….….….….….….….….….….…

# File lib/kaitai/struct/struct.rb, line 239
def read_u2le
  read_bytes(2).unpack('v')[0]
end
read_u4be() click to toggle source
# File lib/kaitai/struct/struct.rb, line 220
def read_u4be
  read_bytes(4).unpack('N')[0]
end
read_u4le() click to toggle source
# File lib/kaitai/struct/struct.rb, line 243
def read_u4le
  read_bytes(4).unpack('V')[0]
end
read_u8be() click to toggle source
# File lib/kaitai/struct/struct.rb, line 225
def read_u8be
  read_bytes(8).unpack('Q')[0]
end
read_u8le() click to toggle source
# File lib/kaitai/struct/struct.rb, line 248
def read_u8le
  read_bytes(8).unpack('Q')[0]
end
seek(x) click to toggle source

Set stream pointer to designated position. @param x [Fixnum] new position (offset in bytes from the beginning of the stream)

# File lib/kaitai/struct/struct.rb, line 136
def seek(x); @_io.seek(x); end
size() click to toggle source

Get total size of the stream in bytes. @return [Fixnum] size of the stream in bytes

# File lib/kaitai/struct/struct.rb, line 146
def size; @_io.size; end

Private Instance Methods

to_signed(x, mask) click to toggle source
# File lib/kaitai/struct/struct.rb, line 527
def to_signed(x, mask)
  (x & ~mask) - (x & mask)
end