good_job

GoodJob

GoodJob is a multithreaded, Postgres-based, Active Job backend for Ruby on Rails.

Inspired by Delayed::Job and Que, GoodJob is designed for maximum compatibility with Ruby on Rails, Active Job, and Postgres to be simple and performant for most workloads.

• Designed for Active Job. Complete support for async, queues, delays, priorities, timeouts, and retries with near-zero configuration.
• Built for Rails. Fully adopts Ruby on Rails threading and code execution guidelines with Concurrent::Ruby.
• Backed by Postgres. Relies upon Postgres integrity, session-level Advisory Locks to provide run-once safety and stay within the limits of schema.rb, and LISTEN/NOTIFY to reduce queuing latency.
• Fully featured. Includes support for cron-like scheduled jobs, batches, concurrency and throttling controls, and a powerful Web Dashboard (check out the Demo).
• Flexible and lightweight. Safely runnable within a single existing web process or scaled via an independent CLI process across development, test, and production environments.
• For most workloads. Targets full-stack teams, economy-minded solo developers, and applications that enqueue 1-million jobs/day and more.

For more of the story of GoodJob, read the introductory blog post.

Comparison of GoodJob with other job queue backends (click to expand)

	Queues, priority, retries	Database	Concurrency	Reliability/Integrity	Latency
GoodJob	✅ Yes	✅ Postgres	✅ Multithreaded	✅ ACID, Advisory Locks	✅ Postgres LISTEN/NOTIFY
Solid Queue	✅ Yes	✅ Postgres and other databases ✨	Multithreaded in forked process	✅ ACID, Advisory Locks	Polling
Que	✅ Yes	️ Postgres, requires structure.sql	✅ Multithreaded	✅ ACID, Advisory Locks	✅ Postgres LISTEN/NOTIFY
Delayed Job	✅ Yes	✅ Postgres	Single-threaded	✅ ACID, record-based	Polling
Sidekiq	✅ Yes	Redis	✅ Multithreaded	Crashes lose jobs	✅ Redis BRPOP
Sidekiq Pro	✅ Yes	Redis	✅ Multithreaded	✅ Redis RPOPLPUSH	✅ Redis RPOPLPUSH

Table of contents

• Set up
• Compatibility
• Configuration
  • Command-line options
    • good_job start
    • good_job cleanup_preserved_jobs
  • Configuration options
  • Global options
  • Dashboard
    • API-only Rails applications
    • Live polling
    • Extending dashboard views
  • Job priority
  • Concurrency controls
    • How concurrency controls work
  • Cron-style repeating/recurring jobs
  • Bulk enqueue
  • Batches
  • Updating
    • Upgrading minor versions
    • Upgrading v3 to v4
    • Upgrading v2 to v3
    • Upgrading v1 to v2
• Go deeper
  • Exceptions, retries, and reliability
    • Exceptions
    • Retries
    • Action Mailer retries
    • Interrupts, graceful shutdown, and SIGKILL
  • Timeouts
  • Optimize queues, threads, and processes
  • Database connections
  • Production setup
  • Queue performance with Queue Select Limit
  • Execute jobs async / in-process
  • Migrate to GoodJob from a different Active Job backend
  • Monitor and preserve worked jobs
  • Write tests
  • PgBouncer compatibility
  • CLI HTTP health check probes
  • Pausing Jobs
• Doing your best job with GoodJob
  • Sizing jobs: mice and elephants
  • Isolating by total latency
  • Configuring your queues
  • Additional observations
• Contribute
  • Gem development
    • Development setup
    • Rails development harness
    • Running tests
  • Release
• License

Set up

1. Add good_job to your application's Gemfile and install the gem:

   bundle add good_job
   

2. Run the GoodJob install generator. This will generate a database migration to create a table for GoodJob's job records:

   bin/rails g good_job:install
   

   Run the migration:

   bin/rails db:migrate
   

   Optional: If using Rails' multiple databases with the migrations_paths configuration option, use the --database option:

   bin/rails g good_job:install --database animals
   bin/rails db:migrate:animals
   

3. Configure the Active Job adapter:

   # config/application.rb or config/environments/{RAILS_ENV}.rb
   config.active_job.queue_adapter = :good_job
   

4. Inside of your application, queue your job :

   YourJob.perform_later
   

   GoodJob supports all Active Job features:

   YourJob.set(queue: :some_queue, wait: 5.minutes, priority: 10).perform_later
   

5. In Rails' development environment, by default, GoodJob's Adapter executes jobs async in a background thread pool in rails server.

   • Because of Rails deferred autoloading, jobs enqueued via the rails console may not begin executing on a separate server process until the Rails application is fully initialized by loading a web page once. To force early initialization (so pre-existing jobs run immediately on boot), add the following to an initializer:

     # config/initializers/good_job.rb
     Rails.configuration.after_initialize do
       ActiveJob::Base && ActiveRecord::Base
     end
     

   • Remember, only Active Job's perform_later sends jobs to the queue adapter; Active Job's perform_now executes the job immediately and does not invoke the queue adapter. GoodJob is not involved in perform_now jobs.

6. In Rails' test environment, by default, GoodJob's Adapter executes jobs inline immediately in the current thread.

   • Future-scheduled jobs can be executed with GoodJob.perform_inline using a tool like Timecop or ActiveSupport::Testing::TimeHelpers.
   • Note that Active Job's TestAdapter, which powers test helpers (e.g. assert_enqueued_with), may override GoodJob's Adapter in some configurations.

7. In Rails' production environment, by default, GoodJob's Adapter enqueues jobs in external mode to be executed by a separate execution process:

   • By default, GoodJob separates job enqueuing from job execution so that jobs can be scaled independently of the web server. Use the GoodJob command-line tool to execute jobs:

     bundle exec good_job start
     

     Ideally the command-line tool should be run on a separate machine or container from the web process. For example, on Heroku:

     web: rails server
     worker: bundle exec good_job start
     

     The command-line tool supports a variety of options, see the reference below for command-line configuration.

   • GoodJob can also be configured to execute jobs within the web server process to save on resources. This is useful for low-workloads when economy is paramount.

     GOOD_JOB_EXECUTION_MODE=async rails server
     

     Additional configuration is likely necessary, see the reference below for configuration.

Compatibility

• Ruby on Rails: 6.1+
• Ruby: Ruby 3.0+. JRuby 9.4+
• Postgres: 10.0+

Configuration

Command-line options

There are several top-level commands available through the good_job command-line tool.

Configuration options are available with help.

good_job start

good_job start executes queued jobs.

…

good_job cleanup_preserved_jobs

good_job cleanup_preserved_jobs destroys preserved job records. See GoodJob.preserve_job_records for when this command is useful.

$ bundle exec good_job help cleanup_preserved_jobs

Usage:
  good_job cleanup_preserved_jobs

Options:
  [--before-seconds-ago=SECONDS] # Destroy records finished more than this many seconds ago (env var:  GOOD_JOB_CLEANUP_PRESERVED_JOBS_BEFORE_SECONDS_AGO, default: 1209600 (14 days))

Manually destroys preserved job records.

By default, GoodJob automatically destroys job records when the job is performed
and this command is not required to be used.

Configuration options

Active Job configuration depends on where the code is placed:

• config.active_job.queue_adapter = :good_job within config/application.rb or config/environments/*.rb.
• ActiveJob::Base.queue_adapter = :good_job within an initializer (e.g. config/initializers/active_job.rb).

GoodJob configuration can be placed within Rails config directory for all environments (config/application.rb), within a particular environment (e.g. config/environments/development.rb), or within an initializer (e.g. config/initializers/good_job.rb).

Configuration examples:

…

Available configuration options are:

• execution_mode (symbol) specifies how and where jobs should be executed. You can also set this with the environment variable GOOD_JOB_EXECUTION_MODE. It can be any one of:
  • :inline executes jobs immediately in whatever process queued them (usually the web server process). This should only be used in test and development environments.
  • :external causes the adapter to enqueue jobs, but not execute them. When using this option (the default for production environments), you’ll need to use the command-line tool to actually execute your jobs.
  • :async (or :async_server) executes jobs in separate threads within the Rails web server process (bundle exec rails server). It can be more economical for small workloads because you don’t need a separate machine or environment for running your jobs, but if your web server is under heavy load or your jobs require a lot of resources, you should choose :external instead. When not in t