Beanpicker

What is it?

Beanpicker is a job queueing DSL for Beanstalk

What? Beanstalk? It make coffe?

beanstalk(d) is a fast, lightweight queueing backend inspired by memcached. The Ruby Beanstalk client is a bit raw, however, so Beanpicker provides a thin wrapper to make job queueing from your Ruby app easy and fun.

Is this similar to Stalker?

Yes, it is inspired in stalker and in Minion

Why should I use Beanpicker instead of Stalker?

Beanpicker work with subprocess. It create a fork for every request and destroy it in the end.

The vantages of Beanpicker are:

How is his performance?

My machine:

The speed with 10000 requests given ‘with fork’ is :fork => :every and ‘without fork’ is :fork => :master/false

# time / requests per second / cpu load

Fork is activated by default, it should slow down your application but keep safe from memory leaks.

You can easy active or desactive the fork for a job with:

job "job.without.fork", :fork => false do |args|
  debug "Running on a thread in main process"
  warn "This process will grow because of any job running on main process"
end

job "job.with.fork.every.time", :fork => :every do |args|
  debug "Running on a fork of main process"
  debug "This process will be killed on end of this job"
  debug "This decrease the peformance but save from memory leaks"
  debug "All extra memory used by this process will vanish in end"
end

job "job.with.fork.once", :fork => :master do |args|
  debug "Running on a fork of main process"
  debug "This process will not be killed on end of this job"
  debug "This increase the performance but don't save from memory leaks"
  debug "This process will only grow in memory because of code executed in 'job.with.fork.once'"
end

You can pass :fork_every => true(default)/false and :fork_master => true/false(default)

The :fork argument overwrite :fork_every and :fork_master

The default :fork_every and :fork_master are setted on Beanpicker::default_fork_

Beanpicker::fork_every and Beanpicker::fork_master overwrite the job options, so, if you set they false the jobs will run in the main thread even if they specify the :fork, :fork_every and/or :fork_master

Queueing jobs

From anywhere in your app:

require 'beanpicker'

Beanpicker.enqueue('email.send', :to => 'joe@example.com')
Beanpicker.enqueue('post.cleanup.all')
Beanpicker.enqueue('post.cleanup', :id => post.id)

Chain jobs

If you have a task that requires more than one step just pass an array of queues when you enqueue.

require 'beanpicker/job_server'

Beanpicker::Worker.new do
  # this is a slow job, so we'll spawn 10 forks of it :)
  job "email.fetch_attachments", :childs => 10 do |args|
    attachment_ids = Email.fetch_attachments_for args[:email_id]
    { :attachment_ids => attachment_ids }
  end

  # by default :childs is 1
  job "email.send" do |args|
    Email.send({
      :id => args[:email_id],
      :attachments => args[:attachment_ids].map { |a| Attachment.find(a) }
    })
  end

end

Beanpicker.enqueue(["email.fetch_attachments", "email.send"], :email_id => 10)

Output Messages

Inside of a job you can use debug, info, warn, error and fatal. It will be redirected to logger(STDOUT by default)

Options

Global options

All options are inside of module Beanpicker

Using combine

Beanpicker ships with “combine”, “A Beanpicker server”

Try combine –help to see all options

e.g. command:

combine -r config.rb -l log/jobs.log sandwich_jobs.rb email_jobs.rb

Multiple Beanstalk servers

Beanpicker look in ENV variables BEANSTALK_URL and BEANSTALK_URLS.

In BEANSTALK_URL it expect a url like “server” or “beanstalk://server”.

In BEANSTALK_URLS it expect a list of urls separed by comma. e.g. “localhost,localhost:11301,10.1.1.9,10.1.1.10:3000”

Credits

Created by Renan Fernandes

Released under the MIT License