class Shrine::Storage::S3

Constants

COPY_OPTIONS
MAX_MULTIPART_PARTS
MIN_PART_SIZE
MULTIPART_THRESHOLD

Attributes

bucket[R]
client[R]
copy_options[R]
prefix[R]
public[R]
signer[R]
upload_options[R]

Public Class Methods

new(bucket:, client: nil, prefix: nil, upload_options: {}, multipart_threshold: {}, max_multipart_parts: nil, signer: nil, public: nil, copy_options: COPY_OPTIONS, **s3_options) click to toggle source

Initializes a storage for uploading to S3. All options are forwarded to [‘Aws::S3::Client#initialize`], except the following:

:bucket : (Required). Name of the S3 bucket.

:client : By default an ‘Aws::S3::Client` instance is created internally from

additional options, but you can use this option to provide your own
client. This can be an `Aws::S3::Client` or an
`Aws::S3::Encryption::Client` object.

:prefix : “Directory” inside the bucket to store files into.

:upload_options : Additional options that will be used for uploading files, they will

be passed to [`Aws::S3::Object#put`], [`Aws::S3::Object#copy_from`]
and [`Aws::S3::Bucket#presigned_post`].

:copy_options : Additional options that will be used for copying files, they will

be passed to [`Aws::S3::Object#copy_from`].

:multipart_threshold : If the input file is larger than the specified size, a parallelized

multipart will be used for the upload/copy. Defaults to
`{upload: 15*1024*1024, copy: 100*1024*1024}` (15MB for upload
requests, 100MB for copy requests).

:max_multipart_parts : Limits the number of parts if parellized multipart upload/copy is used. Defaults to 10_000.

In addition to specifying the ‘:bucket`, you’ll also need to provide AWS credentials. The most common way is to provide them directly via ‘:access_key_id`, `:secret_access_key`, and `:region` options. But you can also use any other way of authentication specified in the [AWS SDK documentation][configuring AWS SDK].

[‘Aws::S3::Object#put`]: docs.aws.amazon.com/sdk-for-ruby/v3/api/Aws/S3/Object.html#put-instance_method [`Aws::S3::Object#copy_from`]: docs.aws.amazon.com/sdk-for-ruby/v3/api/Aws/S3/Object.html#copy_from-instance_method [`Aws::S3::Bucket#presigned_post`]: docs.aws.amazon.com/sdk-for-ruby/v3/api/Aws/S3/Object.html#presigned_post-instance_method [`Aws::S3::Client#initialize`]: docs.aws.amazon.com/sdk-for-ruby/v3/api/Aws/S3/Client.html#initialize-instance_method [configuring AWS SDK]: docs.aws.amazon.com/sdk-for-ruby/v3/developer-guide/setup-config.html

# File lib/shrine/storage/s3.rb, line 71
def initialize(bucket:, client: nil, prefix: nil, upload_options: {}, multipart_threshold: {}, max_multipart_parts: nil, signer: nil, public: nil, copy_options: COPY_OPTIONS, **s3_options)
  raise ArgumentError, "the :bucket option is nil" unless bucket

  @client = client || Aws::S3::Client.new(**s3_options)
  @bucket = Aws::S3::Bucket.new(name: bucket, client: @client)
  @prefix = prefix
  @upload_options = upload_options
  @copy_options = copy_options
  @multipart_threshold = MULTIPART_THRESHOLD.merge(multipart_threshold)
  @max_multipart_parts = max_multipart_parts || MAX_MULTIPART_PARTS
  @signer = signer
  @public = public
end

Public Instance Methods

clear!(&block) click to toggle source

If block is given, deletes all objects from the storage for which the block evaluates to true. Otherwise deletes all objects from the storage.

s3.clear!
# or
s3.clear! { |object| object.last_modified < Time.now - 7*24*60*60 }
# File lib/shrine/storage/s3.rb, line 220
def clear!(&block)
  objects_to_delete = bucket.objects(prefix: prefix)
  objects_to_delete = objects_to_delete.lazy.select(&block) if block

  delete_objects(objects_to_delete)
end
delete(id) click to toggle source

Deletes the file from the storage.

# File lib/shrine/storage/s3.rb, line 200
def delete(id)
  object(id).delete
end
delete_prefixed(delete_prefix) click to toggle source

Deletes objects at keys starting with the specified prefix.

s3.delete_prefixed("somekey/derivatives/")
# File lib/shrine/storage/s3.rb, line 207
def delete_prefixed(delete_prefix)
  # We need to make sure to combine with storage prefix, and
  # that it ends in '/' cause S3 can be squirrely about matching interior.
  delete_prefix = delete_prefix.chomp("/") + "/"
  bucket.objects(prefix: [*prefix, delete_prefix].join("/")).batch_delete!
end
exists?(id) click to toggle source

Returns true file exists on S3.

# File lib/shrine/storage/s3.rb, line 128
def exists?(id)
  object(id).exists?
end
object(id) click to toggle source

Returns an ‘Aws::S3::Object` for the given id.

# File lib/shrine/storage/s3.rb, line 228
def object(id)
  bucket.object(object_key(id))
end
open(id, rewindable: true, encoding: nil, **options) click to toggle source

Returns a ‘Down::ChunkedIO` object that downloads S3 object content on-demand. By default, read content will be cached onto disk so that it can be rewinded, but if you don’t need that you can pass ‘rewindable: false`. A required character encoding can be passed in `encoding`; the default is `Encoding::BINARY` via `Down::ChunkedIO`.

Any additional options are forwarded to [‘Aws::S3::Object#get`].

[‘Aws::S3::Object#get`]: docs.aws.amazon.com/sdk-for-ruby/v3/api/Aws/S3/Object.html#get-instance_method

# File lib/shrine/storage/s3.rb, line 119
def open(id, rewindable: true, encoding: nil, **options)
  chunks, length = get(id, **options)

  Down::ChunkedIO.new(chunks: chunks, rewindable: rewindable, size: length, encoding: encoding)
rescue Aws::S3::Errors::NoSuchKey
  raise Shrine::FileNotFound, "file #{id.inspect} not found on storage"
end
presign(id, method: :post, **presign_options) click to toggle source

Returns URL, params, headers, and verb for direct uploads.

s3.presign("key") #=>
# {
#   url: "https://my-bucket.s3.amazonaws.com/...",
#   fields: { ... },  # blank for PUT presigns
#   headers: { ... }, # blank for POST presigns
#   method: "post",
# }

By default it calls [‘Aws::S3::Object#presigned_post`] which generates data for a POST request, but you can also specify `method: :put` for PUT uploads which calls [`Aws::S3::Object#presigned_url`].

s3.presign("key", method: :post) # for POST upload (default)
s3.presign("key", method: :put)  # for PUT upload

Any additional options are forwarded to the underlying AWS SDK method.

[‘Aws::S3::Object#presigned_post`]: docs.aws.amazon.com/sdk-for-ruby/v3/api/Aws/S3/Object.html#presigned_post-instance_method [`Aws::S3::Object#presigned_url`]: docs.aws.amazon.com/sdk-for-ruby/v3/api/Aws/S3/Object.html#presigned_url-instance_method

# File lib/shrine/storage/s3.rb, line 189
def presign(id, method: :post, **presign_options)
  options = {}
  options[:acl] = "public-read" if public

  options.merge!(@upload_options)
  options.merge!(presign_options)

  send(:"presign_#{method}", id, options)
end
upload(io, id, shrine_metadata: {}, **upload_options) click to toggle source

If the file is an UploadedFile from S3, issues a COPY command, otherwise uploads the file. For files larger than ‘:multipart_threshold` a multipart upload/copy will be used for better performance and more resilient uploads.

It assigns the correct “Content-Type” taken from the MIME type, because by default S3 sets everything to “application/octet-stream”.

# File lib/shrine/storage/s3.rb, line 92
def upload(io, id, shrine_metadata: {}, **upload_options)
  content_type, filename = shrine_metadata.values_at("mime_type", "filename")

  options = {}
  options[:content_type] = content_type if content_type
  options[:content_disposition] = ContentDisposition.inline(filename) if filename
  options[:acl] = "public-read" if public

  options.merge!(@upload_options)
  options.merge!(upload_options)

  if copyable?(io)
    copy(io, id, **options)
  else
    put(io, id, **options)
  end
end
url(id, public: self.public, host: nil, **options) click to toggle source

Returns the presigned URL to the file.

:host : This option replaces the host part of the returned URL, and is

typically useful for setting CDN hosts (e.g.
`http://abc123.cloudfront.net`)

:public : Returns the unsigned URL to the S3 object. This requires the S3

object to be public.

All other options are forwarded to [‘Aws::S3::Object#presigned_url`] or [`Aws::S3::Object#public_url`].

[‘Aws::S3::Object#presigned_url`]: docs.aws.amazon.com/sdk-for-ruby/v3/api/Aws/S3/Object.html#presigned_url-instance_method [`Aws::S3::Object#public_url`]: docs.aws.amazon.com/sdk-for-ruby/v3/api/Aws/S3/Object.html#public_url-instance_method

# File lib/shrine/storage/s3.rb, line 148
def url(id, public: self.public, host: nil, **options)
  if public || signer
    url = object(id).public_url(**options)
  else
    url = object(id).presigned_url(:get, **options)
  end

  if host
    uri = URI.parse(url)
    uri.path = uri.path.match(/^\/#{bucket.name}/).post_match unless uri.host.include?(bucket.name)
    url = URI.join(host, uri.request_uri[1..-1]).to_s
  end

  if signer
    url = signer.call(url, **options)
  end

  url
end

Private Instance Methods

copy(io, id, **copy_options) click to toggle source

Copies an existing S3 object to a new location. Uses multipart copy for large files.

# File lib/shrine/storage/s3.rb, line 247
def copy(io, id, **copy_options)
  # don't inherit source object metadata or AWS tags
  options = {
    metadata_directive: "REPLACE",
  }

  if io.size && io.size >= @multipart_threshold[:copy]
    # pass :content_length on multipart copy to avoid an additional HEAD request
    options.merge!(multipart_copy: true, content_length: io.size)
  end

  options.merge!(@copy_options)
  options.merge!(copy_options)

  object(id).copy_from(io.storage.object(io.id), **options)
end
copyable?(io) click to toggle source

The file is copyable if it’s on S3 and on the same Amazon account.

# File lib/shrine/storage/s3.rb, line 336
def copyable?(io)
  io.is_a?(UploadedFile) &&
  io.storage.is_a?(Storage::S3) &&
  io.storage.client.config.access_key_id == client.config.access_key_id
end
delete_objects(objects) click to toggle source

Deletes all objects in fewest requests possible (S3 only allows 1000 objects to be deleted at once).

# File lib/shrine/storage/s3.rb, line 344
def delete_objects(objects)
  objects.each_slice(1000) do |objects_batch|
    delete_params = { objects: objects_batch.map { |object| { key: object.key } } }
    bucket.delete_objects(delete: delete_params)
  end
end
get(id, **params) click to toggle source
# File lib/shrine/storage/s3.rb, line 304
def get(id, **params)
  enum = object(id).enum_for(:get, **params)

  begin
    content_length = Integer(enum.peek.last["content-length"])
  rescue StopIteration
    content_length = 0
  end

  chunks = Enumerator.new { |y| loop { y << enum.next.first } }

  [chunks, content_length]
end
object_key(id) click to toggle source

Returns object key with potential prefix.

# File lib/shrine/storage/s3.rb, line 352
def object_key(id)
  [*prefix, id].join("/")
end
part_size(io) click to toggle source

Determins the part size that should be used when uploading the given IO object via multipart upload.

# File lib/shrine/storage/s3.rb, line 290
def part_size(io)
  return unless io.respond_to?(:size) && io.size

  if io.size <= MIN_PART_SIZE * @max_multipart_parts # <= 50 GB
    MIN_PART_SIZE
  else # > 50 GB
    (io.size.to_f / @max_multipart_parts).ceil
  end
end
presign_post(id, options) click to toggle source

Generates parameters for a POST upload request.

# File lib/shrine/storage/s3.rb, line 265
def presign_post(id, options)
  presigned_post = object(id).presigned_post(options)

  { method: :post, url: presigned_post.url, fields: presigned_post.fields }
end
presign_put(id, options) click to toggle source

Generates parameters for a PUT upload request.

# File lib/shrine/storage/s3.rb, line 272
def presign_put(id, options)
  url = object(id).presigned_url(:put, options)

  # When any of these options are specified, the corresponding request
  # headers must be included in the upload request.
  headers = {}
  headers["Content-Length"]      = options[:content_length]      if options[:content_length]
  headers["Content-Type"]        = options[:content_type]        if options[:content_type]
  headers["Content-Disposition"] = options[:content_disposition] if options[:content_disposition]
  headers["Content-Encoding"]    = options[:content_encoding]    if options[:content_encoding]
  headers["Content-Language"]    = options[:content_language]    if options[:content_language]
  headers["Content-MD5"]         = options[:content_md5]         if options[:content_md5]

  { method: :put, url: url, headers: headers }
end
put(io, id, **options) click to toggle source

Uploads the file to S3. Uses multipart upload for large files.

# File lib/shrine/storage/s3.rb, line 235
def put(io, id, **options)
  if io.respond_to?(:size) && io.size && io.size <= @multipart_threshold[:upload]
    object(id).put(body: io, **options)
  else # multipart upload
    object(id).upload_stream(part_size: part_size(io), **options) do |write_stream|
      IO.copy_stream(io, write_stream)
    end
  end
end