solid_queue

Solid Queue

Solid Queue is a database-based queuing backend for Active Job, designed with simplicity and performance in mind.

In addition to regular job enqueuing and processing, Solid Queue supports delayed jobs, concurrency controls, recurring jobs, pausing queues, numeric priorities per job, priorities by queue order, and bulk enqueuing (enqueue_all for Active Job's perform_all_later).

Solid Queue can be used with SQL databases such as MySQL, PostgreSQL, or SQLite, and it leverages the FOR UPDATE SKIP LOCKED clause, if available, to avoid blocking and waiting on locks when polling jobs. It relies on Active Job for retries, discarding, error handling, serialization, and delays, and it's compatible with Ruby on Rails's multi-threading.

Table of Contents

• Installation
  • Usage in development and other non-production environments
  • Single database configuration
  • Dashboard UI Setup
  • Incremental adoption
  • High performance requirements
• Workers, dispatchers, and scheduler
  • Fork vs. async mode
• Configuration
  • Optional scheduler configuration
  • Queue order and priorities
  • Queues specification and performance
  • Threads, processes, and signals
  • Database configuration
  • Other configuration settings
  • Validating the configuration
• Lifecycle hooks
• Errors when enqueuing
• Concurrency controls
  • Performance considerations
• Failed jobs and retries
  • Error reporting on jobs
  • Jobs interrupted by non-graceful process death
• Batch jobs
  • Batch progress and counters
  • Batch maintenance
  • Clearing batches
  • Upgrading existing installations
• Puma plugin
• Jobs and transactional integrity
• Recurring tasks
  • Scheduling and unscheduling recurring tasks dynamically
• Inspiration
• License

Installation

Solid Queue is configured by default in new Rails 8 applications. If you're running an earlier version, you can add it manually following these steps:

1. bundle add solid_queue
2. bin/rails solid_queue:install

(Note: The minimum supported version of Rails is 7.1 and Ruby is 3.2.)

This will configure Solid Queue as the production Active Job backend, create the configuration files config/queue.yml and config/recurring.yml, and create the db/queue_schema.rb. It'll also create a bin/jobs executable wrapper that you can use to start Solid Queue.

Once you've done that, you will have to add the configuration for the queue database in config/database.yml. If you're using SQLite, it'll look like this:

production:
  primary:
    <<: *default
    database: storage/production.sqlite3
  queue:
    <<: *default
    database: storage/production_queue.sqlite3
    migrations_paths: db/queue_migrate

...or if you're using MySQL/PostgreSQL/Trilogy:

production:
  primary: &primary_production
    <<: *default
    database: app_production
    username: app
    password: <%= ENV["APP_DATABASE_PASSWORD"] %>
  queue:
    <<: *primary_production
    database: app_production_queue
    migrations_paths: db/queue_migrate

Then run db:prepare in production to ensure the database is created and the schema is loaded.

Now you're ready to start processing jobs by running bin/jobs on the server that's doing the work. This will start processing jobs in all queues using the default configuration. See below to learn more about configuring Solid Queue.

For small projects, you can run Solid Queue on the same machine as your webserver. When you're ready to scale, Solid Queue supports horizontal scaling out-of-the-box. You can run Solid Queue on a separate server from your webserver, or even run bin/jobs on multiple machines at the same time. Depending on the configuration, you can designate some machines to run only dispatchers or only workers. See the configuration section for more details on this.

Note: Future changes to the schema will come in the form of regular migrations.

Usage in development and other non-production environments

Calling bin/rails solid_queue:install will automatically add config.solid_queue.connects_to = { database: { writing: :queue } } to config/environments/production.rb. In order to use Solid Queue in other environments (such as development or staging), you'll need to add a similar configuration(s).

For example, if you're using SQLite in development, update database.yml as follows:

development:
+ primary:
    <<: *default
    database: storage/development.sqlite3
+ queue:
+   <<: *default
+   database: storage/development_queue.sqlite3
+   migrations_paths: db/queue_migrate

Next, add the following to development.rb

  # Use Solid Queue in Development.
  config.active_job.queue_adapter = :solid_queue
  config.solid_queue.connects_to = { database: { writing: :queue } }

Once you've added this, run db:prepare to create the Solid Queue database and load the schema.

Finally, in order for jobs to be processed, you'll need to have Solid Queue running. In Development, this can be done via the Puma plugin as well. In puma.rb update the following line:

# You can either set the env var, or check for development
plugin :solid_queue if ENV["SOLID_QUEUE_IN_PUMA"] || Rails.env.development?

You can also just use bin/jobs, but in this case you might want to set a different logger for Solid Queue because the default logger will log to log/development.log and you won't see anything when you run bin/jobs. For example:

config.solid_queue.logger = ActiveSupport::Logger.new(STDOUT)

Note about Action Cable: If you use Action Cable (or anything dependent on Action Cable, such as Turbo Streams), you will also need to update it to use a database.

In config/cable.yml

development:
-  adapter: async
+  adapter: solid_cable
+  connects_to:
+    database:
+      writing: cable
+  polling_interval: 0.1.seconds
+  message_retention: 1.day

In config/database.yml

development:
  primary:
    <<: *default
    database: storage/development.sqlite3
+  cable:
+    <<: *default
+    database: storage/development_cable.sqlite3
+    migrations_paths: db/cable_migrate

Single database configuration

Running Solid Queue in a separate database is recommended, but it's also possible to use one single database for both the app and the queue. Follow these steps:

1. Copy the contents of db/queue_schema.rb into a normal migration and delete db/queue_schema.rb
2. Remove config.solid_queue.connects_to from production.rb
3. Migrate your database. You are ready to run bin/jobs

You won't have multiple databases, so database.yml doesn't need to have primary and queue database.

Dashboard UI Setup

For viewing information about your jobs via a UI, we recommend taking a look at mission_control-jobs, a dashboard where, among other things, you can examine and retry/discard failed jobs.

Incremental adoption

If you're planning to adopt Solid Queue incrementally by switching one job at the time, you can do so by leaving the config.active_job.queue_adapter set to your old backend, and then set the queue_adapter directly in the jobs you're moving:

# app/jobs/my_job.rb

class MyJob < ApplicationJob
  self.queue_adapter = :solid_queue
  # ...
end

High performance requirements

Solid Queue was designed for the highest throughput when used with MySQL 8+, MariaDB 10.6+, or PostgreSQL 9.5+, as they support FOR UPDATE SKIP LOCKED. You can use it with older versions, but in that case, you might run into lock waits if you run multiple workers for the same queue. You can also use it with SQLite on smaller applications.

Workers, dispatchers, and scheduler

We have several types of actors in Solid Queue:

• Workers are in charge of picking jobs ready to run from queues and processing them. They work off the solid_queue_ready_executions table.
• Dispatchers are in charge of selecting jobs scheduled to run in the future that are due and dispatching them, which is simply moving them from the solid_queue_scheduled_executions table over to the solid_queue_ready_executions table so that workers can pick them up. On top of that, they do some maintenance work related to concurrency controls.
• The scheduler manages recurring tasks, enqueuing jobs for them when they're due.
• The supervisor runs workers and dispatchers according to the configuration, controls their heartbeats, and stops and starts them when needed.

Fork vs. async mode

By default, Solid Queue runs in fork mode. This means the supervisor will fork a separate process for each supervised worker/dispatcher/scheduler. This provides the best isolation and performance, but can have additional memory usage and might not work with some Ruby implementations. As an alternative, you can run all workers, dispatchers and schedulers in the same process as the supervisor, in different threads, with an async mode. You can choose this mode by running bin/jobs as:

bin/jobs --mode async

Or you can also set the environment variable SOLID_QUEUE_SUPERVISOR_MODE to async. If you use the async mode, the processes option in the configuration described below will be ignored.

The recommended and default mode is fork. Only use async if you know what you're doing and have strong reasons to

This supervisor mode is separate from a worker's concurrency model. Supervisor mode decides whether supervised processes live in forks or threads. Worker configuration decides whether claimed jobs run in a thread pool (threads: N) or as fibers on a single fiber reactor thread (fibers: N).

Because these are separate concerns, you can combine the default fork supervisor mode with fiber workers. In that setup, each worker process gets its own fiber reactor and bounded fiber count.

Configuration

By default, Solid Queue will try to find your configuration under config/queue.yml, but you can set a different path using the environment variable SOLID_QUEUE_CONFIG or by using the -c/--config_file option with bin/jobs, like this:

bin/jobs -c config/calendar.yml

You can also skip the scheduler process by setting the environment variable SOLID_QUEUE_SKIP_RECURRING=true. This is useful for environments like staging, review apps, or development where you don't want any recurring jobs to run. This is equivalent to using the --skip-recurring option with bin/jobs.

To run only the scheduler (no workers or dispatchers)—for example to isolate recurring tasks on a dedicated process—set SOLID_QUEUE_ONLY_RECURRING=true or use the --only-recurring option with bin/jobs.

This is what this configuration looks like:

production:
  dispatchers:
    - polling_interval