Queue / Pipeline Management
Qless is a powerful Redis-based job queueing system inspired by
resque,
but built on a collection of Lua scripts, maintained in the
qless-core repo.
A job is a unit of work identified by a job id or jid. A queue can contain
several jobs that are scheduled to be run at a certain time, several jobs that are
waiting to run, and jobs that are currently running. A worker is a process on a
host, identified uniquely, that asks for jobs from the queue, performs some process
associated with that job, and then marks it as complete. When it's completed, it
can be put into another queue.
Jobs can only be in one queue at a time. That queue is whatever queue they were last put in. So if a worker is working on a job, and you move it, the worker's request to complete the job will be ignored.
A job can be canceled, which means it disappears into the ether, and we'll never
pay it any mind ever again. A job can be dropped, which is when a worker fails
to heartbeat or complete the job in a timely fashion, or a job can be failed,
which is when a host recognizes some systematically problematic state about the
job. A worker should only fail a job if the error is likely not a transient one;
otherwise, that worker should just drop it and let the system reclaim it.
qless automatically keeps statistics about how long jobs wait
to be processed and how long they take to be processed. Currently, we keep
track of the count, mean, standard deviation, and a histogram of these times.First things first, require qless and create a client. The client accepts all the
same arguments that you'd use when constructing a redis client.
require 'qless'
# Connect to localhost
client = Qless::Client.new
# Connect to somewhere else
client = Qless::Client.new(:host => 'foo.bar.com', :port => 1234)Jobs should be classes or modules that define a perform method, which
must accept a single job argument:
class MyJobClass
def self.perform(job)
# job is an instance of `Qless::Job` and provides access to
# job.data, a means to cancel the job (job.cancel), and more.
end
endNow you can access a queue, and add a job to that queue.
# This references a new or existing queue 'testing'
queue = client.queues['testing']
# Let's add a job, with some data. Returns Job ID
queue.put(MyJobClass, :hello => 'howdy')
# => "0c53b0404c56012f69fa482a1427ab7d"
# Now we can ask for a job
job = queue.pop
# =>
# And we can do the work associated with it!
job.performThe job data must be serializable to JSON, and it is recommended that you use a hash for it. See below for a list of the supported job options.
The argument returned by queue.put is the job ID, or jid. Every Qless
job has a unique jid, and it provides a means to interact with an
existing job:
…The Qless ruby worker was heavily inspired by Resque's worker, but thanks to the power of the qless-core lua scripts, it is much simpler and you are welcome to write your own (e.g. if you'd rather save memory by not forking the worker for each job).
As with resque...
RUN_AS_SINGLE_PROCESS
environment variable to force Qless to not fork the child process.
Single process mode should only be used in some test/dev
environments.ps.VERBOSE or VVERBOSE (very
verbose) environment variables.qless:work) for running workers.
It runs qless:setup before starting the main work loop so that
users can load their environment in that task.INTERVAL environment variable.Resque uses queues for its notion of priority. In contrast, qless has priority support built-in. Thus, the worker supports two strategies for what order to pop jobs off the queues: ordered and round-robin. The ordered reserver will keep popping jobs off the first queue until it is empty, before trying to pop job off the second queue. The round-robin reserver will pop a job off the first queue, then the second queue, and so on. You could also easily implement your own.
To start a worker, write a bit of Ruby code that instantiates a worker and runs it. You could write a rake task to do this, for example:
…The following signals are supported in the parent process:
You should send these to the master process, not the child.
The child process supports the USR2 signal, whch causes it to
dump its current backtrace.
Workers also support middleware modules that can be used to inject logic before, after or around the processing of a single job in the child process. This can be useful, for example, when you need to re-establish a connection to your database in each job.
Define a module with an around_perform method that calls super where you
want the job to be processed:
module ReEstablishDBConnection
def around_perform(job)
MyORM.establish_connection
super
end
endThen, mix-it into the worker class. You can mix-in as many middleware modules as you like:
require 'qless/worker'
Qless::Worker.class_eval do
include ReEstablishDBConnection
include SomeOtherAwesomeMiddleware
endQless also supports middleware on a per-job basis, when you have some orthogonal logic to run in the context of some (but not all) jobs.
Per-job middlewares are defined the same as worker middlewares:
module ReEstablishDBConnection
def around_perform(job)
MyORM.establish_connection
super
end
endTo add them to a job class, you first have to make your job class
middleware-enabled by extending it with
Qless::Job::SupportsMiddleware, then extend your middleware
modules:
class MyJobClass
extend Qless::Job::SupportsMiddleware
extend ReEstablishDBConnection
extend MyOtherAwesomeMiddleware
def self.perform(job)
end
endNote that Qless::Job::SupportsMiddleware must be extended onto your
job class before any other middleware modules.
Qless ships with a resque-inspired web app that lets you easily
deal with failures and see what it is processing. If you're project
has a rack-based ruby web app, we recommend you mount Qless's web app
in it. Here's how you can do that with Rack::Builder in your config.ru:
client = Qless::Client.new(:host => "some-host", :port => 7000)
Rack::Builder.new do
use SomeMiddleware
map('/some-other-app') { run Apps::Something.new }
map('/qless') { run Qless::Server.new(client) }
endFor an app using Rails 3+, check the router documentation for how to mount rack apps.
If you wish to run the web interface from the exe directory, you
have the option to run the server as a daemon. Running as a daemon is
default behavior. To run in the foreground, pass the --foreground or
-F flag:
PATH_TO_QLESS_DIST/exe/qless-web -FLet's say you have one job that depends on another, but the task definitions are fundamentally different. You need to bake a turkey, and you need to make stuffing, but you can't make the turkey until the stuffing is made:
queue = client.queues['cook']
stuffing_jid = queue.put(MakeStuffing, {:lots => 'of butter'})
turkey_jid = queue.put(MakeTurkey , {:with => 'stuffing'}, :depends=>[stuffing_jid])When the stuffing job completes, the turkey job is unlocked and free to be processed.
Some jobs need to get popped sooner than others. Whether it's a trouble ticket, or debugging, you can do this pretty easily when you put a job in a queue:
queue.put(MyJobClass, {:foo => 'bar'}, :priority => 10)What happens when you want to adjust a job's priority while it's still waiting in a queue?
job = client.jobs['0c53b0404c56012f69fa482a1427ab7d']
job.priority = 10
# Now this will get popped before any job of lower priorityIf you don't want a job to be run right away but some time in the future, you can specify a delay:
# Run at least 10 minutes from now
queue.put(MyJobClass, {:foo => 'bar'}, :delay => 600)This doesn't guarantee that job will be run exactly at 10 minutes. You can accomplish this by changing the job's priority so that once 10 minutes has elapsed, it's put before lesser-priority jobs:
# Run in 10 minutes
queue.put(MyJobClass, {:foo => 'bar'}, :delay => 600, :priority => 100)Sometimes it's not enough simply to schedule one job, but you want to run jobs regularly. In particular, maybe you have some batch operation that needs to get run once an hour and you don't care what worker runs it. Recurring jobs are specified much like other jobs:
# Run every hour
queue.recur(MyJobClass, {:widget => 'warble'}, 3600)
# => 22ac75008a8011e182b24cf9ab3a8f3bYou can even access them in much the same way as you would normal jobs:
job = client.jobs['22ac75008a8011e182b24cf9ab3a8f3b']
# => Changing the interval at which it runs after the fact is trivial:
# I think I only need it to run once every two hours
job.interval = 7200If you want it to run every hour on the hour, but it's 2:37 right now, you can specify an offset which is how long it should wait before popping the first job:
# 23 minutesNo open issues yet, or sync has not completed.