Elastictastic
¶ ↑
Elastictastic
is an object-document mapper and lightweight API adapter for ElasticSearch. Elastictastic’s primary use case is to define model classes which use ElasticSearch as a primary document-oriented data store, and to expose ElasticSearch’s search functionality to query for those models.
Dependencies¶ ↑
Elastictastic
requires Ruby 1.9 and ActiveSupport 3. Elastictastic
does not require Rails, but if you do run Rails, Elastictastic
will only work with Rails 3.
You will also need a running ElasticSearch instance (or cluster). For local development, you can easily download and install a copy, or your preferred package manager might have it available.
Installation¶ ↑
Just add it to your Gemfile:
gem 'elastictastic'
Defining models¶ ↑
Elastictastic’s setup DSL will be familiar to those who have used other Ruby object-document mappers such as Mongoid. Persisted models mix in the Elastictastic::Document
module, and fields are defined with the field
class macro:
class Post include Elastictastic::Document field :title end
The field
method can take options; the options available here are simply those that are available in a field mapping in ElasticSearch. Elastictastic
is (mostly) agnostic to the options you pass in; they’re just used to generate the mapping for ElasticSearch.
By default, ElasticSearch assigns fields a string
type. An example of how one might define a field with some options:
class Post include Elastictastic::Document field :comments_count, :type => :integer, :store => 'yes' end
Multi-fields¶ ↑
ElasticSearch allows you to define multi-fields, which index the same data in multiple ways. To define a multi-field in Elastictastic
, you may pass a block to the field
macro, in which the alternate fields are defined using the same DSL:
field :title, :type => 'string', :index => 'analyzed' do field :unanalyzed, :type => 'string', :index => 'not_analyzed' end
The arguments passed to the outer field
method are used for the default field mapping; thus, the above is the same as the following:
field :title, :type => 'multi_field', :fields => { :title => { :type => 'string', :index => 'analyzed' }, :unanalyzed => { :type => 'string', :index => 'not_analyzed' } }
Document Boost¶ ↑
Defining a document boost will increase or decrease a document’s score in search results based on the value of a field in the document. A boost of 1.0 is neutral. To define a boost field, use the boost
class macro:
class Post include Elastictastic::Document field :score, :type => 'integer' boost :score end
By default, if the boost field is empty, a score of 1.0 will be applied. You can override this by passing a 'null_value'
option into the boost method.
Embedded objects¶ ↑
ElasticSearch supports deep nesting of properties by way of object fields. To define embedded objects in your Elastictastic
models, use the embed
class macro:
class Post include Elastictastic::Document embed :author embed :recent_comments, :class_name => 'Comment' end
The class that’s embedded should include the Elastictastic::NestedDocument
mixin, which exposes the same configuration DSL as Elastictastic::Document
but does not give the class the functionality of a top-level persistent object:
class Author include Elastictastic::NestedDocument field :name field :email, :index => 'not_analyzed' end
Parent-child relationships¶ ↑
You may define parent-child relationships for your documents using the has_many
and belongs_to
macros:
class Blog include Elastictastic::Document has_many :posts end
class Post include Elastictastic::Document belongs_to :blog end
Unlike in, say, ActiveRecord, an Elastictastic
document can only specify one parent (belongs_to
) relationship. A document can have as many children (has_many
) as you would like.
The parent/child relationship has far-reaching consequences in ElasticSearch, and as such you will generally interact with child documents via the parent’s association collection. For instance, this is the standard way to create a new child instance:
post = blog.posts.new
The above will return a new Post object whose parent is the blog
; the blog.posts
collection will retain a reference to the transient post
instance, and will auto-save it when the blog
is saved.
You may also create a child instance independently and then add it to a parent’s child collection; however, you must do so before saving the child instance, as ElasticSeach requires types that define parents to have a parent. The following code block has the same outcome as the previous one:
post = Post.new blog.posts << post
In most other respects, the blog.posts
collection behaves the same as a search scope (more on that below), except that enumeration methods (#each
, #map
, etc.) will return unsaved child instances along with instances persisted in ElasticSearch.
Syncing your mapping¶ ↑
Before you start creating documents with Elastictastic
, you need to make ElasticSearch aware of your document structure. To do this, use the sync_mapping
method:
Post.sync_mapping
If you have a complex multi-index topology, you may want to consider using ElasticSearch templates to manage mappings and other index settings; Elastictastic
doesn’t provide any explicit support for this at the moment, although you can use e.g. Post.mapping
to retrieve the mapping structure which you can then merge into your template.
Reserved attributes¶ ↑
All Elastictastic::Document
models have an id
and an index
field, which combine to define the full resource locator for the document in ElasticSearch. You should not define fields or methods with these names. You may, however, set the id explicitly on new (not yet saved) model instances.
ActiveModel¶ ↑
Elastictastic
documents include all the usual ActiveModel functionality: validations, lifecycle hooks, observers, dirty-tracking, mass-assignment security, and the like. If you would like to squeeze a bit of extra performance out of the library at the cost of convenience, you can include the Elastictastic::BasicDocument
module instead of Elastictastic::Document
.
Persistence¶ ↑
Elastictastic
models are persisted the usual way, namely by calling save
:
post = Post.new post.title = 'You know, for search.' post.save
To retrieve a document from the data store, use find
:
Post.find('123')
You can look up multiple documents by ID:
Post.find('123', '456')
You can also pass an array of IDs; the following will return a one-element array:
Post.find(['123'])
For child documents, you must perform GET requests using the parent’s association collection:
post = blog.posts.new post.save blog.posts.find(post.id) # this will return the post Post.find(post.id) # but this won't!
Specifying the index¶ ↑
Elastictastic
defines a default index for your documents. If you’re using Rails, the default index is your application’s name suffixed with the current environment; outside of Rails, the default index is simply “default”. You can change this using the default_index
configuration key.
When you want to work with documents in an index other than the default, use the in_index
class method:
new_post = Post.in_index('my_special_index').new # create in an index post = Post.in_index('my_special_index').find('123') # retrieve from an index
To retrieve documents from multiple indices at the same time, pass a hash into find
where the keys are index names and the values are the IDs you wish to retrieve from that index:
Post.find('default' => ['123', '456'], 'my_special_index' => '789')
Bulk operations¶ ↑
If you are writing a large amount of data to ElasticSearch in a single process, use of the bulk API is encouraged. To perform bulk operations using Elastictastic
, simply wrap your operations in a bulk
block:
Elastictastic.bulk do params[:posts].each do |post_params| post = Post.new(post_params) post.save end end
All create, update, and destroy operations inside the block will be executed in a single bulk request when the block completes. If you are performing an indefinite number of operations in a bulk block, you can pass an :auto_flush
option to flush the bulk buffer after the specified number of operations:
Elastictastic.bulk(:auto_flush => 100) do 150.times { Post.new.save! } end
The above will perform two bulk requests: the first after the first 100 operations, and the second when the block completes.
Note that the nature of bulk writes means that any operation inside a bulk block is essentially asynchronous: instances are not created, updated, or destroyed immediately upon calling save
or destroy
, but rather when the bulk block exits. You may pass a block to save
and destroy
to provide a callback for when the instance is actually persisted and its local state updated. Let’s say, for instance, we wish to expand the example above to pass the IDs of the newly created posts to our view layer:
@ids = [] Elastictastic.bulk do params[:posts].each do |post_params| post = Post.new(post_params) post.save do |e| @ids << post.id end end end
If the save was not successful (due to a duplicate ID or a version mismatch, for instance), the e
argument to the block will be passed an exception object; if the save was successful, the argument will be nil.
Concurrent document creation¶ ↑
When Elastictastic
creates a document with an application-defined ID, it uses the _create
verb in ElasticSearch, ensuring that a document with that ID does not already exist. If the document does already exist, an Elastictastic::ServerError::DocumentAlreadyExistsEngineException
will be raised. In the case where multiple processes may attempt concurrent creation of the same document, you can gracefully handle concurrent creation using the ::create_or_update
class method on your document class. This will first attempt to create the document; if a document with that ID already exists, it will then load the document and modify it using the block passed:
Post.create_or_update('1') do |post| post.title = 'My Post' end
In the above case, Elastictastic
will first attempt to create a new post with ID “1” and title “My Post”. If a Post with that ID already exists, it will load it, set its title to “My Post”, and save it. The update uses the ::update
method (see next section) to ensure that concurrent modification doesn’t cause data to be lost.
Optimistic locking¶ ↑
Elastictastic
provides optimistic locking via ElasticSearch’s built-in document versioning. When a document is retrieved from persistence, it carries a version, which is a number that increments from 1 on each update. When Elastictastic
models are updated, the document version that it carried when it was loaded is passed into the update operation; if this version does not match ElasticSearch’s current version for that document, it indicates that another process has modified the document concurrently, and an Elastictastic::ServerError::VersionConflictEngineException
is raised. This prevents data loss through concurrent conflicting updates.
The easiest way to guard against concurrent modification is to use the ::update
class method to make changes to existing documents. Consider the following example:
Post.update('12345') do |post| post.title = 'New Title' end
In the above, the Post with ID ‘12345’ is loaded from ElasticSearch and yielded to the block. When the block completes, the instance is saved back to ElasticSearch. If this save results in a version conflict, a new instance is loaded from ElasticSearch and the block is run again. The process repeats until a successful update.
This method will work inside a bulk operation, but note that if the first update generates a version conflict, additional updates will occur in discrete requests, not as part of any bulk operation.
If you wish to safely update documents retrieved from a search scope (see below), use the update_each
method:
Post.query { constant_score { filter { term(:blog_id => 1) }}}.update_each do |post| post.title = post.title.upcase end
Search¶ ↑
ElasticSearch is, above all, a search tool. Accordingly, aside from direct lookup by ID, all retrieval of documents is done via the search API. Elastictastic
models have class methods corresponding to the top-level keys in the ElasticSearch search API; you may chain these much as in ActiveRecord or Mongoid:
Post.query(:query_string => { :query => 'pizza' }).facets(:cuisine => { :term => { :field => :tags }}).from(10).size(10) # Generates {"query": {"query_string": {"query": "pizza"}}, "facets": {"cuisine": {"term": {"field": "tags" }}}, "from": 10, "size": 10}
Elastictastic
also has an alternate block-based query builder, if you prefer:
Post.query do query_string { query('pizza') } end.facets { cuisine { term { field :tags }}}.from(10).size(10) # Same effect as the previous example
The scopes that are generated by the preceding calls act as collections of matching documents; thus all the usual Enumerable methods are available:
Post.query(:query_string => { :query => 'pizza' }).each do |post| puts post.title end
You may access other components of the response using hash-style access; this will return a Hashie::Mash
which allows hash-style or object-style access:
Post.facets(:cuisine => { :term => { :field => :tags }})['facets'].each_pair do |name, facet| facet.terms.each { |term| puts "#{term.term}: #{term.count}" } end
You can also call count
on a scope; this will give the total number of documents matching the query.
In some situations, you may wish to access metadata about search results beyond simply the result document. To do this, use the #find_each
method, which yields a Hashie::Mash
containing the raw ElasticSearch hit object in the second argument:
Post.highlight { fields(:title => {}) }.find_each do |post, hit| puts "Post #{post.id} matched the query string in the title field: #{hit.highlight['title']}" end
Search scopes also expose a #find_in_batches
method, which also yields the raw hit. The following code gives the same result as the previous example:
Post.highlight { fields(:title => {}) }.find_in_batches do |batch| batch.each do |post, hit| puts "Post #{post.id} matched the query string in the title field: #{hit.highlight['title']}" end end
Both find_each
and find_in_batches
accept a :batch_size
option.
Support & Bugs¶ ↑
If you find a bug, feel free to open an issue on GitHub. Pull requests are most welcome.
For questions or feedback, hit up our mailing list at elastictastic@groups.google.com or find outoftime on the elasticsearch IRC channel on Freenode.
License¶ ↑
Elastictastic
is distributed under the MIT license. See the attached LICENSE file for all the sordid details.