适用于 Ruby 的简单可靠的 Beanstalkd 任务队列
Backburner is a beanstalkd-powered job queue that can handle a very high volume of jobs. You create background jobs and place them on multiple work queues to be processed later.
Processing background jobs reliably has never been easier than with beanstalkd and Backburner. This gem works with any ruby-based web framework, but is especially suited for use with Sinatra, Padrino and Rails.
If you want to use beanstalk for your job processing, consider using Backburner. Backburner is heavily inspired by Resque and DelayedJob. Backburner stores all jobs as simple JSON message payloads. Persistent queues are supported when beanstalkd persistence mode is enabled.
Backburner supports multiple queues, job priorities, delays, and timeouts. In addition, Backburner has robust support for retrying failed jobs, handling error cases, custom logging, and extensible plugin hooks.
Backburner is well tested and has a familiar, no-nonsense approach to job processing, but that is of secondary importance. Let's face it, there are a lot of options for background job processing. DelayedJob, and Resque are the first that come to mind immediately. So, how do we make sense of which one to use? And why use Backburner over other alternatives?
The key to understanding the differences lies in understanding the different projects and protocols that power these popular queue libraries under the hood. Every job queue requires a queue store that jobs are put into and pulled out of. In the case of Resque, jobs are processed through Redis, a persistent key-value store. In the case of DelayedJob, jobs are processed through ActiveRecord and a database such as PostgreSQL.
The work queue underlying these gems tells you infinitely more about the differences than anything else. Beanstalk is probably the best solution for job queues available today for many reasons. The real question then is... "Why Beanstalk?".
Illya has an excellent blog post Scalable Work Queues with Beanstalk and Adam Wiggins posted an excellent comparison.
You will quickly see that beanstalkd is an underrated but incredible project that is extremely well-suited as a job queue. Significantly better suited for this task than Redis or a database. Beanstalk is a simple, and a very fast work queue service rolled into a single binary - it is the memcached of work queues. Originally built to power the backend for the 'Causes' Facebook app, it is a mature and production ready open source project. PostRank uses beanstalk to reliably process millions of jobs a day.
A single instance of Beanstalk is perfectly capable of handling thousands of jobs a second (or more, depending on your job size) because it is an in-memory, event-driven system. Powered by libevent under the hood, it requires zero setup (launch and forget, à la memcached), optional log based persistence, an easily parsed ASCII protocol, and a rich set of tools for job management that go well beyond a simple FIFO work queue.
Beanstalkd supports the following features out of the box:
| Feature | Description |
|---|---|
| Parallelized | Supports multiple work queues created on demand. |
| Reliable | Beanstalk’s reserve, work, delete cycle ensures reliable processing. |
| Scheduling | Delay enqueuing jobs by a specified interval to schedule processing later |
| Fast | Processes thousands of jobs per second without breaking a sweat. |
| Priorities | Specify priority so important jobs can be processed quickly. |
| Persistence | Jobs are stored in memory for speed, but logged to disk for safe keeping. |
| Federation | Horizontal scalability provided through federation by the client. |
| Error Handling | Bury any job which causes an error for later debugging and inspection. |
Keep in mind that these features are supported out of the box with beanstalk and require no special code within this gem to support. In the end, beanstalk is the ideal job queue while also being ridiculously easy to install and setup.
First, you probably want to install beanstalkd, which powers the job queues. Depending on your platform, this should be as simple as (for Ubuntu):
$ sudo apt-get install beanstalkdAdd this line to your application's Gemfile:
gem 'backburner'And then execute:
$ bundleOr install it yourself as:
$ gem install backburnerBackburner is extremely simple to setup. Just configure basic settings for backburner:
…The key options available are:
| Option | Description |
|---|---|
beanstalk_url |
Address for beanstalkd connection i.e 'beanstalk://127.0.0.1' |
tube_namespace |
Prefix used for all tubes related to this backburner queue. |
namespace_separator |
Separator used for namespace and queue name |
on_error |
Lambda invoked with the error whenever any job in the system fails. |
max_job_retries |
Integer defines how many times to retry a job before burying. |
retry_delay |
Integer defines the base time to wait (in secs) between job retries. |
retry_delay_proc |
Lambda calculates the delay used, allowing for exponential back-off. |
default_priority |
Integer The default priority of jobs |
respond_timeout |
Integer defines how long a job has to complete its task |
default_worker |
Worker class that will be used if no other worker is specified. |
logger |
Logger recorded to when backburner wants to report info or errors. |
primary_queue |
Primary queue used for a job when an alternate queue is not given. |
priority_labels |
Hash of named priority definitions for your app. |
reserve_timeout |
Duration to wait for work from a single server, or nil for forever. |
job_serializer_proc |
Lambda serializes a job body to a string to write to the task |
job_parser_proc |
Lambda parses a task body string to a hash |
Before v0.4.0: Jobs were placed into default queues based on the name of the class creating the queue. i.e NewsletterJob would be put into a 'newsletter-job' queue. As of 0.4.0, all jobs are placed into a primary queue named "my.app.namespace.backburner-jobs" unless otherwise specified.
Backburner allows you to create jobs and place them onto any number of beanstalk tubes, and later pull those jobs off the tubes and process them asynchronously with a worker.
At the core, Backburner is about jobs that can be processed asynchronously. Jobs are simple ruby objects which respond to perform.
Job objects are queued as JSON onto a tube to be later processed by a worker. Here's an example:
…You can include the optional Backburner::Queue module so you can easily specify queue settings for this job:
class NewsletterJob
include Backburner::Queue
queue "newsletter-sender" # defaults to 'backburner-jobs' tube
queue_priority 1000 # most urgent priority is 0
queue_respond_timeout 300 # number of seconds before job times out, 0 to avoid timeout
def self.perform(email, body)
NewsletterMailer.deliver_text_to_email(email, body)
end
endJobs can be enqueued with:
Backburner.enqueue NewsletterJob, '[email protected]', 'lorem ipsum...'Backburner.enqueue accepts first a ruby object that supports perform and then a series of parameters
to that object's perform method. The queue name used by default is {namespace}.backburner-jobs
unless otherwise specified.
You may also pass a lambda as the queue name and it will be evaluated when enqueuing a job (and passed the Job's class as an argument). This is especially useful when combined with "Simple Async Jobs" (see below).
In addition to defining custom jobs, a job can also be enqueued by invoking the async method on any object which
includes Backburner::Performable. Async enqueuing works for both instance and class methods on any performable object.
…This automatically enqueues a job for that user record that will run activate with the specified argument.
Note that you can set the queue name and queue priority at the class level and
you are also able to pass pri, ttr, delay and queue directly as options into async.
The queue name used by default is {namespace}.backburner-jobs if not otherwise
specified.
If a lambda is given for queue, then it will be called and given the
performable object's class as an argument:
# Given the User class above
User.async(:queue => lambda { |user_klass| ["queue1","queue2"].sample(1).first }).do_hard_work # would add the job to either queue1 or queue2 randomlyIt's often useful to be able to configure your app in production such that every invocation of a method is asynchronous by default as seen in delayed_job. To accomplish this, the Backburner::Performable module exposes two handle_asynchronously convenience methods
which accept the same options as the async method:
class User
include Backburner::Performable
def send_welcome_email
# ...
end
# ---> For instance methods
handle_asynchronously :send_welcome_email, queue: 'send-mail', pri: 5000, ttr: 60
def self.update_recent_visitors
# ...
end
# ---> For class methods
handle_static_asynchronously :update_recent_visitors, queue: 'long-tasks', ttr: 300
endNow, all calls to User.update_recent_visitors or User#send_welcome_email will automatically be handled asynchronously when invoked. Similarly, you can call these methods directly on the Backburner::Performable module to apply async behavior outside the class:
# Given the User class above
Backburner::Performable.handle_asynchronously(User, :activate, ttr: 100, queue: 'activate')Now all calls to the activate method on a User instance will be async with the provided options.
Because an async proxy is injected and used in place of the original method, you must not rely on the return value of the method. Using the example User class above, if my send_welcome_email returned the status of an email submission and I relied on that to take some further action, I will be surprised after rewiring things with handle_asynchronously because the async proxy actually returns the (boolean) result of Backburner::Worker.enqueue.
Backburner workers are processes that run forever handling jobs that are reserved from the queue. Starting a worker in ruby code is simple:
Backburner.workThis will process jobs in all queues but you can also restrict processing to specific queues:
Backburner.work('newsletter-sender', 'push-notifier')The Backburner worker also exists as a rake task:
require 'backburner/tasks'so you can run:
$ QUEUE=newsletter-sender,push-notifier rake backburner:workYou can also run the backburner binary for a c
暂无开放 Issues,或尚未同步最近议题。