module NexosisApi::Client::Imports

Imports-based API operations

@see docs.nexosis.com/

Public Instance Methods

import_from_azure(dataset_name, connection_string, container, blob_name, column_metadata = []) click to toggle source

Import a csv, json file (gzip'd or raw) from a Microsoft Azure storage blob

@param dataset_name [String] the name to give to the new dataset or existing dataset to which this data will be upserted @param connection_string [String] the azure blob storage connection string providing access to the file resource @param container [String] the container in which the object is located. @param blob_name [String] the name of the object to import, usually a file. Always csv or json content. If folders have been used this will contain the full path within the container. @param column_metadata [Array of NexosisApi::Column] description of each column in target dataset. Optional. @return [NexosisApi::ImportsResponse] @since 2.0.0 @note the connection string provided will be encrypted at the server, used once, and then removed from storage.

# File lib/nexosis_api/client/imports.rb, line 85
def import_from_azure(dataset_name, connection_string, container, blob_name, column_metadata = [])
  raise ArgumentError, 'dataset_name was not provided and is not optional ' unless dataset_name.empty? == false
  raise ArgumentError, 'connection_string was not provided and is not optional ' unless connection_string.empty? == false
  raise ArgumentError, 'container was not provided and is not optional ' unless container.empty? == false
  raise ArgumentError, 'blob_name was not provided and is not optional ' unless blob_name.empty? == false
  azure_url = '/imports/azure'
  column_json = Column.to_json(column_metadata)
  body = {
    'dataSetName' => dataset_name,
    'connectionString' => connection_string,
    'container' => container,
    'blob' => blob_name,
    'columns' => column_json
  }
  response = self.class.post(azure_url, headers: @headers, body: body.to_json)
  raise HttpException.new("There was a problem importing from azure: #{response.code}.",
                          "uploading dataset from azure #{dataset_name}",
                          response) unless response.success?
  NexosisApi::ImportsResponse.new(response.parsed_response)
end
import_from_s3(dataset_name, bucket_name, path, region = 'us-east-1', credentials = {}, column_metadata = []) click to toggle source

Import a file from AWS s3 as your dataset

@param dataset_name [String] the name to give to the new dataset or existing dataset to which this data will be upserted @param bucket_name [String] the AWS S3 bucket name in which the path will be found @param path [String] the path within the bucket (usually file name) @param region [String] the region in which your bucket exists. Defaults to us-east-1 @param credentials [Hash] :access_key_id and :secret_access_key for user with rights to read the target file. @param column_metadata [Array of NexosisApi::Column] description of each column in target dataset. Optional. @return [NexosisApi::ImportsResponse] @see docs.aws.amazon.com/general/latest/gr/rande.html#s3_region for information on region names @note If credentials are provided they will be encrypted at the server, used once, and then removed from storage.

# File lib/nexosis_api/client/imports.rb, line 37
def import_from_s3(dataset_name, bucket_name, path, region = 'us-east-1', credentials = {}, column_metadata = [])
  raise ArgumentError, 'dataset_name was not provided and is not optional ' unless dataset_name.to_s.empty? == false
  raise ArgumentError, 'bucket_name was not provided and is not optional ' unless bucket_name.to_s.empty? == false
  raise ArgumentError, 'path was not provided and is not optional ' unless path.to_s.empty? == false
  s3_import_url = '/imports/s3'
  column_json = Column.to_json(column_metadata)
  body = {
    'dataSetName' => dataset_name,
    'bucket' => bucket_name,
    'path' => path,
    'region' => region,
    'columns' => column_json
  }
  body['accessKeyId'] = credentials[:access_key_id] unless credentials.nil? || credentials[:access_key_id].nil?
  body['secretAccessKey'] = credentials[:secret_access_key] unless credentials.nil? || credentials[:secret_access_key].nil?
  response = self.class.post(s3_import_url, headers: @headers, body: body.to_json)
  raise HttpException.new("There was a problem importing from s3: #{response.code}.",
                          "uploading dataset from s3 #{dataset_name}",
                          response) unless response.success?
  NexosisApi::ImportsResponse.new(response.parsed_response)
end
import_from_url(dataset_name, url, column_metadata = [], options = {}) click to toggle source

Import a csv or json file directly from any avaiable and reachable public endpoint.

@param dataset_name [String] the name to give to the new dataset or existing dataset to which this data will be upserted @param url [String] the url indicating where to find the file resource to import @param column_metadata [Array of NexosisApi::Column] description of each column in target dataset. Optional. @param options [Hash] may provide basic auth credentials or a 'content-type' value to identify csv or json content. @return [NexosisApi::ImportsResponse] @note imports depend on file extensions, so use a content type indicator if json or csv cannot be inferred. @note Urls protected by basic auth can be accessed if given a userid and password in options @since 2.0.0

# File lib/nexosis_api/client/imports.rb, line 116
def import_from_url(dataset_name, url, column_metadata = [], options = {})
  raise ArgumentError, 'dataset_name was not provided and is not optional ' unless dataset_name.empty? == false
  raise ArgumentError, 'url was not provided and is not optional ' unless url.empty? == false
  endpoint_url = '/imports/url'
  column_json = Column.to_json(column_metadata)
  body = {
    'dataSetName' => dataset_name,
    'url' => url,
    'columns' => column_json
  }
  response = self.class.post(endpoint_url, headers: @headers, body: body.to_json)
  raise HttpException.new("There was a problem importing from url: #{response.code}.",
                          "uploading dataset from #{url}",
                          response) unless response.success?
  NexosisApi::ImportsResponse.new(response.parsed_response)
end
list_imports(import_list_query = NexosisApi::ImportListQuery.new) click to toggle source

List all existing import requests

@param import_list_query [NexosisApi::ImportListQuery] options to limit the query results @return [NexosisApi::PagedArray of NexosisApi::ImportsResponse] @since 1.4 added paging parameters

# File lib/nexosis_api/client/imports.rb, line 14
def list_imports(import_list_query = NexosisApi::ImportListQuery.new)
  imports_url = '/imports'
  response = self.class.get(imports_url, headers: @headers, query: import_list_query.query_parameters)
  if (response.success?)
    NexosisApi::PagedArray.new(response.parsed_response,
                               response.parsed_response['items']
                               .map { |i| NexosisApi::ImportsResponse.new(i) })
  else
    raise HttpException.new("There was a problem getting the imports: #{response.code}.", "listing imports with query #{import_list_query.query_parameters}", response)
  end
end
retrieve_import(import_id) click to toggle source

Get response back from import created previously. Presumably to check status.

@param import_id [String] The id returned from a previous request to import @return [NexosisApi::ImportsResponse] @example get import

NexosisApi.client.retrieve_import('740dca2a-b488-4322-887e-fa473b1caa54')
# File lib/nexosis_api/client/imports.rb, line 65
def retrieve_import(import_id)
  raise ArgumentError, 'import_id was not provided and is not optional ' unless import_id.to_s.empty? == false
  imports_url = "/imports/#{import_id}"
  response = self.class.get(imports_url, headers: @headers)
  raise HttpException.new("There was a problem getting the import #{response.code}.",
                          "requesting an import #{import_id}", response) unless response.success?
  NexosisApi::ImportsResponse.new(response.parsed_response)
end