Streaming tactics **¶
Note
Certain constructs are available only for seekable streams (in-memory and files) and some require tellable streams (which in fact is a subset of seekability). Sockets and pipes do not support seeking, so you’ll have to first read the data from the stream, and parse it in-memory, or use the Rebuffered()
wrapper.
Wrappers¶
Pointer¶
Pointer allows for non-sequential construction. The pointer first changes the stream position, does the construction, and restores the original stream position.
>>> Pointer(8, Bytes(1)).parse(b"abcdefghijkl")
b'i'
>>> Pointer(8, Bytes(1)).build(b"x")
b'\x00\x00\x00\x00\x00\x00\x00\x00x'
Peek¶
Parses the subconstruct but restores the stream position afterwards (it does “peeking”).
>>> Sequence(Peek(Byte), Peek(Int16ub)).parse(b"\x01\x02")
[1, 258]
Pure side effects¶
Seek makes a jump within the stream and leaves it at that point. It does not read or write anything to the stream by itself.
-
class
construct.core.
Seek
(at, whence=0)¶ Sets a new stream position when parsing or building. Seeks are useful when many other fields follow the jump. Pointer works when there is only one field to look at, but when there is more to be done, Seek may come useful.
See also
Analog
Pointer()
wrapper that has same side effect but also processed a subcon.- Parameters
at – where to jump to, can ne an int or a context lambda
whence – is the offset from beginning (0) or from current position (1) or from ending (2), can be an int or a context lambda, default is 0
Example:
>>> (Seek(5) >> Byte).parse(b"01234x") [5, 120] >>> (Bytes(10) >> Seek(5) >> Byte).build([b"0123456789", None, 255]) b'01234\xff6789'
Tell checks the current stream position and returns it, also putting it into the context. It does not read or write anything to the stream by itself.
-
construct.core.
Tell
()¶ Gets the stream position when parsing or building.
Tells are useful for adjusting relative offsets to absolute positions, or to measure sizes of Constructs. To get an absolute pointer, use a Tell plus a relative offset. To get a size, place two Tells and measure their difference using a Compute.
See also
Better to use
CopyRaw()
wrapper in almost any case.Example:
>>> Struct("num"/VarInt, "offset"/Tell).build(dict(num=88)) b'X' >>> Struct("num"/VarInt, "offset"/Tell).parse(_) Container(num=88)(offset=1)
Stream manipulation¶
-
construct.core.
Rebuffered
(subcon, tailcutoff=None)¶ Caches bytes from the underlying stream, so it becomes seekable and tellable. Also makes the stream blocking, in case it came from a socket or a pipe. Optionally, stream can forget bytes that went a certain amount of bytes beyond the current offset, allowing only a limited seeking capability while allowing to process an endless stream.
Warning
Experimental implementation. May not be mature enough.
- Parameters
subcon – the subcon which will operate on the buffered stream
Example:
Rebuffered(RepeatUntil(lambda obj,ctx: ?,Byte), tailcutoff=1024).parse_stream(endless_nonblocking_stream)
-
construct.core.
Restreamed
(subcon, encoder, encoderunit, decoder, decoderunit, sizecomputer)¶ Transforms bytes between the underlying stream and the subcon.
When the parsing or building is done, the wrapper stream is closed. If read buffer or write buffer is not empty, error is raised.
See also
Both
Bitwise()
andBytewise()
are implemented using Restreamed.Warning
Do not use pointers inside.
- Parameters
subcon – the subcon which will operate on the buffer
encoder – a function that takes a b-string and returns a b-string (used when building)
encoderunit – ratio as int, encoder takes that many bytes at once
decoder – a function that takes a b-string and returns a b-string (used when parsing)
decoderunit – ratio as int, decoder takes that many bytes at once
Example:
Bitwise is implemented as Restreamed(subcon, bits2bytes, 8, bytes2bits, 1, lambda n: n//8) Bytewise is implemented as Restreamed(subcon, bytes2bits, 1, bits2bytes, 8, lambda n: n*8)