bes  Updated for version 3.20.8
dmrpp::Chunk Class Reference

#include <Chunk.h>

Collaboration diagram for dmrpp::Chunk:
Collaboration graph

Public Member Functions

void add_tracking_query_param ()
 Modify this chunk's data URL so that it includes tracking info. More...
 
 Chunk ()
 Get an empty chunk. More...
 
 Chunk (const Chunk &h4bs)
 
 Chunk (std::string data_url, std::string order, unsigned long long size, unsigned long long offset, const std::string &pia_str="")
 Get a chunk initialized with values. More...
 
 Chunk (std::string data_url, std::string order, unsigned long long size, unsigned long long offset, const std::vector< unsigned int > &pia_vec)
 Get a chunk initialized with values. More...
 
virtual void dump (std::ostream &strm) const
 
virtual std::string get_byte_order ()
 Get the chunk byte order. More...
 
virtual unsigned long long get_bytes_read () const
 Get the number of bytes read so far for this Chunk. More...
 
virtual std::string get_curl_range_arg_string ()
 Returns a curl range argument. The libcurl requires a string argument for range-ge activitys, this method constructs one in the required syntax from the offset and size information for this byteStream. More...
 
virtual std::string get_data_url () const
 Get the data url string for this Chunk's data block. More...
 
virtual bool get_is_read ()
 
virtual unsigned long long get_offset () const
 Get the offset to this Chunk's data block. More...
 
virtual const std::vector< unsigned int > & get_position_in_array () const
 
virtual char * get_rbuf ()
 
virtual unsigned long long get_rbuf_size () const
 
virtual std::string get_response_content_type ()
 Get the response type of the last response. More...
 
virtual unsigned long long get_size () const
 Get the size of this Chunk's data block on disk. More...
 
virtual void inflate_chunk (bool deflate, bool shuffle, unsigned int chunk_size, unsigned int elem_width)
 Decompress data in the chunk, managing the Chunk's data buffers. More...
 
Chunkoperator= (const Chunk &rhs)
 
virtual void read_chunk ()
 
virtual void set_bytes_read (unsigned long long bytes_read)
 Set the size of this Chunk's data block. More...
 
virtual void set_data_url (const std::string &data_url)
 Set the data url string for this Chunk's data block. More...
 
virtual void set_is_read (bool state)
 
void set_position_in_array (const std::string &pia)
 parse the chunk position string More...
 
void set_position_in_array (const std::vector< unsigned int > &pia)
 Set the chunk's position in the Array. More...
 
virtual void set_rbuf_to_size ()
 Allocates the internal read buffer to be d_size bytes. More...
 
void set_read_buffer (char *buf, unsigned long long buf_size, unsigned long long bytes_read=0, bool assume_ownership=true)
 Set the target read buffer for this chunk. More...
 
void set_response_content_type (const std::string &ct)
 Set the response type of the last response. More...
 
virtual std::string to_string () const
 

Static Public Member Functions

static void parse_chunk_position_in_array_string (const std::string &pia, std::vector< unsigned int > &pia_vect)
 

Protected Member Functions

void _duplicate (const Chunk &bs)
 

Friends

class ChunkTest
 
class DmrppCommonTest
 
class MockChunk
 

Detailed Description

This class is used to encapsulate the state and behavior needed for reading chunked data associated with a DAP variable. In particular it is based on the semantics of an hdf4:Chunk object, which is used to represent a chunk of data in a (potentially complex) HDF4/HDF5 file.

Definition at line 50 of file Chunk.h.

Constructor & Destructor Documentation

◆ Chunk() [1/3]

dmrpp::Chunk::Chunk ( )
inline

Get an empty chunk.

Note
This constructor does not read the Query String marker from the BES context system. You must call Chunk::add_tracking_query_param() if you want that information added with Chunks created using this constructor.
See also
Chunk::add_tracking_query_param()

Definition at line 125 of file Chunk.h.

◆ Chunk() [2/3]

dmrpp::Chunk::Chunk ( std::string  data_url,
std::string  order,
unsigned long long  size,
unsigned long long  offset,
const std::string &  pia_str = "" 
)
inline

Get a chunk initialized with values.

Parameters
data_urlWhere to read this chunk's data
orderThe data storage byte_order
sizeThe number of bytes to read
offsetRead
  • size bytes starting from this offset
pia_strA string that provides the logical position of this chunk in an Array. Has the syntax '[1,2,3,4]'.

Definition at line 142 of file Chunk.h.

◆ Chunk() [3/3]

dmrpp::Chunk::Chunk ( std::string  data_url,
std::string  order,
unsigned long long  size,
unsigned long long  offset,
const std::vector< unsigned int > &  pia_vec 
)
inline

Get a chunk initialized with values.

Parameters
data_urlWhere to read this chunk's data
orderThe data storage byte order
sizeThe number of bytes to read
offsetRead
  • size bytes starting from this offset
pia_vecThe logical position of this chunk in an Array; a std::vector of unsigned ints.

Definition at line 163 of file Chunk.h.

Member Function Documentation

◆ add_tracking_query_param()

void dmrpp::Chunk::add_tracking_query_param ( )

Modify this chunk's data URL so that it includes tracking info.

Add information to the Query string of a URL, intended primarily to aid in tracking the origin of requests when reading data from S3. The information added to the query string comes from a BES Context command sent to the BES by a client (e.g., the OLFS). The addition takes the form "?tracking_context=<context value>".

Note
This is only added to data URLs that reference S3.

Cloudydap test hack where we tag the S3 URLs with a query string for the S3 log in order to track S3 requests. The tag is submitted as a BESContext with the request. Here we check to see if the request is for an AWS S3 object, if it is AND we have the magic BESContext "cloudydap" then we add a query parameter to the S3 URL for tracking purposes.

Should this be a function? FFS why? This is the ONLY place where this needs happen, as close to the curl call as possible and we can just turn it off down the road. - ndp 1/20/17 (EOD)

Well, it's a function now... ;-) jhrg 8/6/18

Definition at line 461 of file Chunk.cc.

◆ dump()

void dmrpp::Chunk::dump ( std::ostream &  strm) const
virtual

unsigned long long d_size; unsigned long long d_offset; std::string d_md5; std::string d_uuid; std::string d_data_url; std::vector<unsigned int> d_chunk_position_in_array;

Definition at line 646 of file Chunk.cc.

◆ get_byte_order()

virtual std::string dmrpp::Chunk::get_byte_order ( )
inlinevirtual

Get the chunk byte order.

Definition at line 205 of file Chunk.h.

◆ get_bytes_read()

virtual unsigned long long dmrpp::Chunk::get_bytes_read ( ) const
inlinevirtual

Get the number of bytes read so far for this Chunk.

Definition at line 239 of file Chunk.h.

◆ get_curl_range_arg_string()

string dmrpp::Chunk::get_curl_range_arg_string ( )
virtual

Returns a curl range argument. The libcurl requires a string argument for range-ge activitys, this method constructs one in the required syntax from the offset and size information for this byteStream.

Definition at line 446 of file Chunk.cc.

◆ get_data_url()

std::string dmrpp::Chunk::get_data_url ( ) const
virtual

Get the data url string for this Chunk's data block.

Definition at line 669 of file Chunk.cc.

◆ get_offset()

virtual unsigned long long dmrpp::Chunk::get_offset ( ) const
inlinevirtual

Get the offset to this Chunk's data block.

Definition at line 218 of file Chunk.h.

◆ get_position_in_array()

virtual const std::vector<unsigned int>& dmrpp::Chunk::get_position_in_array ( ) const
inlinevirtual
Returns
The chunk's position in the array, as a vector of ints.

Definition at line 351 of file Chunk.h.

◆ get_rbuf()

virtual char* dmrpp::Chunk::get_rbuf ( )
inlinevirtual

Returns a pointer to the memory buffer for this Chunk. The return value is NULL if no memory has been allocated.

Definition at line 282 of file Chunk.h.

◆ get_rbuf_size()

virtual unsigned long long dmrpp::Chunk::get_rbuf_size ( ) const
inlinevirtual

Returns the size, in bytes, of the current read buffer for this Chunk.

Definition at line 343 of file Chunk.h.

◆ get_response_content_type()

virtual std::string dmrpp::Chunk::get_response_content_type ( )
inlinevirtual

Get the response type of the last response.

Definition at line 199 of file Chunk.h.

◆ get_size()

virtual unsigned long long dmrpp::Chunk::get_size ( ) const
inlinevirtual

Get the size of this Chunk's data block on disk.

Definition at line 210 of file Chunk.h.

◆ inflate_chunk()

void dmrpp::Chunk::inflate_chunk ( bool  deflate,
bool  shuffle,
unsigned int  chunk_size,
unsigned int  elem_width 
)
virtual

Decompress data in the chunk, managing the Chunk's data buffers.

This method tracks if a chunk has already been decompressed, so, like read_chunk() it can be called for a chunk that has already been decompressed without error.

Parameters
deflateTrue if the chunk should be 'inflated'
shuffleTrue if the chunk should be 'unshuffled'
chunk_sizeThe expected chunk size, in elements; used to allocate storage
elem_widthThe number of bytes per element

Definition at line 530 of file Chunk.cc.

◆ operator=()

Chunk& dmrpp::Chunk::operator= ( const Chunk rhs)
inline

I think this is broken. vector<Chunk> assignment fails in the read_atomic() method but 'assignment' using a reference works. This bug shows up in DmrppCommnon::read_atomic(). jhrg 4/10/18

Definition at line 189 of file Chunk.h.

◆ read_chunk()

void dmrpp::Chunk::read_chunk ( )
virtual

This method is for reading one chunk after the other, using a CURL handle from the CurlHandlePool.

Parameters
deflate
shuffle
chunk_size
elem_width

Definition at line 606 of file Chunk.cc.

◆ set_bytes_read()

virtual void dmrpp::Chunk::set_bytes_read ( unsigned long long  bytes_read)
inlinevirtual

Set the size of this Chunk's data block.

Parameters
sizeSize of the data in bytes

Definition at line 248 of file Chunk.h.

◆ set_data_url()

virtual void dmrpp::Chunk::set_data_url ( const std::string &  data_url)
inlinevirtual

Set the data url string for this Chunk's data block.

Definition at line 231 of file Chunk.h.

◆ set_position_in_array() [1/2]

void dmrpp::Chunk::set_position_in_array ( const std::string &  pia)

parse the chunk position string

Extract information from the chunk position string and store as a vector of integers in the instance

Note
If we can change the DMR++ syntax to be less verbose and use a list of ints with whitespace as a separator, then the parsing code will be much simpler since istringstream is designed to deal with exactly that form of input.
Parameters
piaThe chunk position string. Syntax parsed: "[1,2,3,4]"

Definition at line 395 of file Chunk.cc.

◆ set_position_in_array() [2/2]

void dmrpp::Chunk::set_position_in_array ( const std::vector< unsigned int > &  pia)

Set the chunk's position in the Array.

Use this method when the vector<unsigned int> is known.

See also
Chunk::set_position_in_array(const string &pia)
Parameters
piaA vector of unsigned ints.

Definition at line 431 of file Chunk.cc.

◆ set_rbuf_to_size()

virtual void dmrpp::Chunk::set_rbuf_to_size ( )
inlinevirtual

Allocates the internal read buffer to be d_size bytes.

The memory of the read buffer is managed internally by this method.

The class maintains an internal flag, d_read_buffer_is_mine, which controls if the currently held read buffer memory is released (via a call to 'delete[]') when an this method is invoked.

If the CHunk owns the read buffer, then calling this method will release any previously allocated read buffer memory and then allocate a new memory block. The bytes_read counter is reset to zero.

Definition at line 267 of file Chunk.h.

◆ set_read_buffer()

void dmrpp::Chunk::set_read_buffer ( char *  buf,
unsigned long long  buf_size,
unsigned long long  bytes_read = 0,
bool  assume_ownership = true 
)
inline

Set the target read buffer for this chunk.

Parameters
bufThe new buffer to install into the Chunk.
buf_sizeThe size of the passed buffer.
bytes_readThe number of bytes that have been read into buf. In practice this is the offset in buf at which new bytes should be added, (default: 0)
assume_ownershipIf true, then the memory pointed to by buf will be deleted (using delete[]) when the Chunk object's destructor is called. If false then the Chunk's destructor will not attempt to free/delete the memory pointed to by buf. (default: true)

Definition at line 324 of file Chunk.h.

◆ set_response_content_type()

void dmrpp::Chunk::set_response_content_type ( const std::string &  ct)
inline

Set the response type of the last response.

Definition at line 202 of file Chunk.h.


The documentation for this class was generated from the following files: