百科.dev
全部条目在线工具AI 编程趋势榜开源项目技术资讯提交条目
登录
< 返回工具列表
B

backburner

> 编程语言
开源

适用于 Ruby 的简单可靠的 Beanstalkd 任务队列

433 stars0 点赞3 次浏览
访问官网GitHub

工具介绍

适用于 Ruby 的简单可靠的 Beanstalkd 任务队列

Backburner

Backburner is a beanstalkd-powered job queue that can handle a very high volume of jobs. You create background jobs and place them on multiple work queues to be processed later.

Processing background jobs reliably has never been easier than with beanstalkd and Backburner. This gem works with any ruby-based web framework, but is especially suited for use with Sinatra, Padrino and Rails.

If you want to use beanstalk for your job processing, consider using Backburner. Backburner is heavily inspired by Resque and DelayedJob. Backburner stores all jobs as simple JSON message payloads. Persistent queues are supported when beanstalkd persistence mode is enabled.

Backburner supports multiple queues, job priorities, delays, and timeouts. In addition, Backburner has robust support for retrying failed jobs, handling error cases, custom logging, and extensible plugin hooks.

Why Backburner?

Backburner is well tested and has a familiar, no-nonsense approach to job processing, but that is of secondary importance. Let's face it, there are a lot of options for background job processing. DelayedJob, and Resque are the first that come to mind immediately. So, how do we make sense of which one to use? And why use Backburner over other alternatives?

The key to understanding the differences lies in understanding the different projects and protocols that power these popular queue libraries under the hood. Every job queue requires a queue store that jobs are put into and pulled out of. In the case of Resque, jobs are processed through Redis, a persistent key-value store. In the case of DelayedJob, jobs are processed through ActiveRecord and a database such as PostgreSQL.

The work queue underlying these gems tells you infinitely more about the differences than anything else. Beanstalk is probably the best solution for job queues available today for many reasons. The real question then is... "Why Beanstalk?".

Why Beanstalk?

Illya has an excellent blog post Scalable Work Queues with Beanstalk and Adam Wiggins posted an excellent comparison.

You will quickly see that beanstalkd is an underrated but incredible project that is extremely well-suited as a job queue. Significantly better suited for this task than Redis or a database. Beanstalk is a simple, and a very fast work queue service rolled into a single binary - it is the memcached of work queues. Originally built to power the backend for the 'Causes' Facebook app, it is a mature and production ready open source project. PostRank uses beanstalk to reliably process millions of jobs a day.

A single instance of Beanstalk is perfectly capable of handling thousands of jobs a second (or more, depending on your job size) because it is an in-memory, event-driven system. Powered by libevent under the hood, it requires zero setup (launch and forget, à la memcached), optional log based persistence, an easily parsed ASCII protocol, and a rich set of tools for job management that go well beyond a simple FIFO work queue.

Beanstalkd supports the following features out of the box:

Feature Description
Parallelized Supports multiple work queues created on demand.
Reliable Beanstalk’s reserve, work, delete cycle ensures reliable processing.
Scheduling Delay enqueuing jobs by a specified interval to schedule processing later
Fast Processes thousands of jobs per second without breaking a sweat.
Priorities Specify priority so important jobs can be processed quickly.
Persistence Jobs are stored in memory for speed, but logged to disk for safe keeping.
Federation Horizontal scalability provided through federation by the client.
Error Handling Bury any job which causes an error for later debugging and inspection.

Keep in mind that these features are supported out of the box with beanstalk and require no special code within this gem to support. In the end, beanstalk is the ideal job queue while also being ridiculously easy to install and setup.

Installation

First, you probably want to install beanstalkd, which powers the job queues. Depending on your platform, this should be as simple as (for Ubuntu):

$ sudo apt-get install beanstalkd

Add this line to your application's Gemfile:

gem 'backburner'

And then execute:

$ bundle

Or install it yourself as:

$ gem install backburner

Configuration

Backburner is extremely simple to setup. Just configure basic settings for backburner:

…

The key options available are:

Option Description
beanstalk_url Address for beanstalkd connection i.e 'beanstalk://127.0.0.1'
tube_namespace Prefix used for all tubes related to this backburner queue.
namespace_separator Separator used for namespace and queue name
on_error Lambda invoked with the error whenever any job in the system fails.
max_job_retries Integer defines how many times to retry a job before burying.
retry_delay Integer defines the base time to wait (in secs) between job retries.
retry_delay_proc Lambda calculates the delay used, allowing for exponential back-off.
default_priority Integer The default priority of jobs
respond_timeout Integer defines how long a job has to complete its task
default_worker Worker class that will be used if no other worker is specified.
logger Logger recorded to when backburner wants to report info or errors.
primary_queue Primary queue used for a job when an alternate queue is not given.
priority_labels Hash of named priority definitions for your app.
reserve_timeout Duration to wait for work from a single server, or nil for forever.
job_serializer_proc Lambda serializes a job body to a string to write to the task
job_parser_proc Lambda parses a task body string to a hash

Breaking Changes

Before v0.4.0: Jobs were placed into default queues based on the name of the class creating the queue. i.e NewsletterJob would be put into a 'newsletter-job' queue. As of 0.4.0, all jobs are placed into a primary queue named "my.app.namespace.backburner-jobs" unless otherwise specified.

Usage

Backburner allows you to create jobs and place them onto any number of beanstalk tubes, and later pull those jobs off the tubes and process them asynchronously with a worker.

Enqueuing Jobs

At the core, Backburner is about jobs that can be processed asynchronously. Jobs are simple ruby objects which respond to perform.

Job objects are queued as JSON onto a tube to be later processed by a worker. Here's an example:

…

You can include the optional Backburner::Queue module so you can easily specify queue settings for this job:

ruby
class NewsletterJob
  include Backburner::Queue
  queue "newsletter-sender"  # defaults to 'backburner-jobs' tube
  queue_priority 1000 # most urgent priority is 0
  queue_respond_timeout 300 # number of seconds before job times out, 0 to avoid timeout

  def self.perform(email, body)
    NewsletterMailer.deliver_text_to_email(email, body)
  end
end

Jobs can be enqueued with:

ruby
Backburner.enqueue NewsletterJob, '[email protected]', 'lorem ipsum...'

Backburner.enqueue accepts first a ruby object that supports perform and then a series of parameters to that object's perform method. The queue name used by default is {namespace}.backburner-jobs unless otherwise specified.

You may also pass a lambda as the queue name and it will be evaluated when enqueuing a job (and passed the Job's class as an argument). This is especially useful when combined with "Simple Async Jobs" (see below).

Simple Async Jobs

In addition to defining custom jobs, a job can also be enqueued by invoking the async method on any object which includes Backburner::Performable. Async enqueuing works for both instance and class methods on any performable object.

…

This automatically enqueues a job for that user record that will run activate with the specified argument. Note that you can set the queue name and queue priority at the class level and you are also able to pass pri, ttr, delay and queue directly as options into async.

The queue name used by default is {namespace}.backburner-jobs if not otherwise specified.

If a lambda is given for queue, then it will be called and given the performable object's class as an argument:

ruby
# Given the User class above
User.async(:queue => lambda { |user_klass| ["queue1","queue2"].sample(1).first }).do_hard_work # would add the job to either queue1 or queue2 randomly

Using Async Asynchronously

It's often useful to be able to configure your app in production such that every invocation of a method is asynchronous by default as seen in delayed_job. To accomplish this, the Backburner::Performable module exposes two handle_asynchronously convenience methods which accept the same options as the async method:

ruby
class User
  include Backburner::Performable

  def send_welcome_email
    # ...
  end

  # ---> For instance methods
  handle_asynchronously :send_welcome_email, queue: 'send-mail', pri: 5000, ttr: 60

  def self.update_recent_visitors
    # ...
  end

  # ---> For class methods
  handle_static_asynchronously :update_recent_visitors, queue: 'long-tasks', ttr: 300
end

Now, all calls to User.update_recent_visitors or User#send_welcome_email will automatically be handled asynchronously when invoked. Similarly, you can call these methods directly on the Backburner::Performable module to apply async behavior outside the class:

ruby
# Given the User class above
Backburner::Performable.handle_asynchronously(User, :activate, ttr: 100, queue: 'activate')

Now all calls to the activate method on a User instance will be async with the provided options.

A Note About Auto-Async

Because an async proxy is injected and used in place of the original method, you must not rely on the return value of the method. Using the example User class above, if my send_welcome_email returned the status of an email submission and I relied on that to take some further action, I will be surprised after rewiring things with handle_asynchronously because the async proxy actually returns the (boolean) result of Backburner::Worker.enqueue.

Working Jobs

Backburner workers are processes that run forever handling jobs that are reserved from the queue. Starting a worker in ruby code is simple:

ruby
Backburner.work

This will process jobs in all queues but you can also restrict processing to specific queues:

ruby
Backburner.work('newsletter-sender', 'push-notifier')

The Backburner worker also exists as a rake task:

ruby
require 'backburner/tasks'

so you can run:

$ QUEUE=newsletter-sender,push-notifier rake backburner:work

You can also run the backburner binary for a c

Issues· 48 开放

查看全部 Issues在 GitHub 打开

暂无开放 Issues,或尚未同步最近议题。

> 标签

Ruby

暂无评论,来聊聊你的看法吧

> 工具信息

发布日期2026年8月1日
最后更新2026年9月17日
分类编程语言
定价开源

> 相关工具

T
TypeScript
JavaScript 的超集,为前端与全栈提供静态类型
P
Python
通用编程语言,广泛用于 Web、数据与 AI
G
Go
Google 推出的简洁高效系统语言