maintenance_tasks

Maintenance Tasks

A Rails engine for queuing and managing maintenance tasks.

By ”maintenance task”, this project means a data migration, i.e. code that changes data in the database, often to support schema migrations. For example, in order to introduce a new NOT NULL column, it has to be added as nullable first, backfilled with values, before finally being changed to NOT NULL. This engine helps with the second part of this process, backfilling.

Maintenance tasks are collection-based tasks, usually using Active Record, that update the data in your database. They can be paused or interrupted. Maintenance tasks can operate in batches and use throttling to control the load on your database.

Maintenance tasks aren't meant to happen on a regular basis. They're used as needed, or as one-offs. Normally maintenance tasks are ephemeral, so they are used briefly and then deleted.

The Rails engine has a web-based UI for listing maintenance tasks, seeing their status, and starting, pausing and restarting them.

Should I Use Maintenance Tasks?

Maintenance tasks have a limited, specific job UI. While the engine can be used to provide a user interface for other data changes, such as data changes for support requests, we recommend you use regular application code for those use cases instead. These inevitably require more flexibility than this engine will be able to provide.

If your task shouldn't run as an Active Job, it probably isn't a good match for this gem. If your task doesn't need to run in the background, consider a runner script instead. If your task doesn't need to be interruptible, consider a normal Active Job.

Maintenance tasks can be interrupted between iterations. If your task isn't collection-based (no CSV file or database table) or has very large batches, it will get limited benefit from throttling (pausing between iterations) or interrupting. This might be fine, or the added complexity of maintenance Tasks over normal Active Jobs may not be worthwhile.

If your task updates your database schema instead of data, use a migration instead of a maintenance task.

If your task happens regularly, consider Active Jobs with a scheduler or cron, job-iteration jobs and/or custom rails_admin UIs instead of the Maintenance Tasks gem. Maintenance tasks should be ephemeral, to suit their intentionally limited UI. They should not repeat.

To create seed data for a new application, use the provided Rails db/seeds.rb file instead.

If your application can't handle a half-completed migration, maintenance tasks are probably the wrong tool. Remember that maintenance tasks are intentionally pausable and can be cancelled halfway.

Installation

To install the gem and run the install generator, execute:

bundle add maintenance_tasks
bin/rails generate maintenance_tasks:install

The generator creates and runs a migration to add the necessary table to your database. It also mounts Maintenance Tasks in your config/routes.rb. By default the web UI can be accessed in the new /maintenance_tasks path.

This gem uses the Rails Error Reporter to report errors. If you are using a bug tracking service you may want to subscribe to the reporter. See Reporting Errors for more information.

Active Job Dependency

The Maintenance Tasks framework relies on Active Job behind the scenes to run Tasks. The default queuing backend for Active Job is asynchronous. It is strongly recommended to change this to a persistent backend so that Task progress is not lost during code or infrastructure changes. For more information on configuring a queuing backend, take a look at the Active Job documentation.

Action Controller & Action View Dependency

The Maintenance Tasks framework relies on Action Controller and Action View to render the UI. If you're using Rails in API-only mode, see Using Maintenance Tasks in API-only applications.

Autoloading

The Maintenance Tasks framework does not support autoloading in :classic mode. Please ensure your application is using Zeitwerk to load your code. For more information, please consult the Rails guides on autoloading and reloading constants.

Usage

The typical Maintenance Tasks workflow is as follows:

1. Generate a class describing the Task and the work to be done.
2. Run the Task
   • either by using the included web UI,
   • or by using the command line,
   • or by using Ruby.
3. Monitor the Task
   • either by using the included web UI,
   • or by manually checking your task’s run’s status in your database.
4. Optionally, delete the Task code if you no longer need it.

Creating a Task

A generator is provided to create tasks. Generate a new task by running:

bin/rails generate maintenance_tasks:task update_posts

This creates the task file app/tasks/maintenance/update_posts_task.rb.

The generated task is a subclass of MaintenanceTasks::Task that implements:

• collection: return an Active Record Relation or an Array to be iterated over.
• process: do the work of your maintenance task on a single record

Optionally, tasks can also implement a custom #count method, defining the number of elements that will be iterated over. Your task’s tick_total will be calculated automatically based on the collection size, but this value may be overridden if desired using the #count method (this might be done, for example, to avoid the query that would be produced to determine the size of your collection).

Example:

# app/tasks/maintenance/update_posts_task.rb

module Maintenance
  class UpdatePostsTask (field) { field.strip })

    def process(row)
      Post.create!(title: row["title"], content: row["content"])
    end
  end
end

These options instruct Ruby's CSV parser to skip lines that start with a #, and removes the leading and trailing spaces from any field, so that the following file will be processed identically as the previous example:

posts.csv:

# A comment
title,content
 My Title ,Hello World!

Batch CSV Tasks

Tasks can process CSVs in batches. Add the in_batches option to your task’s csv_collection macro:

# app/tasks/maintenance/batch_import_posts_task.rb

module Maintenance
  class BatchImportPostsTask  { RandomBackoffGenerator.generate_duration } ) do
      DatabaseStatus.unhealthy?
    end
    # ...
  end
end

Custom Task Parameters

Tasks may need additional information, supplied via parameters, to run. Parameters can be defined as Active Model Attributes in a Task, and then become accessible to any of Task’s methods: #collection, #count, or #process.

# app/tasks/maintenance/update_posts_via_params_task.rb

module Maintenance
  class UpdatePostsViaParamsTask  e
  Rails.logger.error(e)
end

ActiveSupport::Notifications.subscribe("errored.maintenance_tasks") do |*, payload|
  task_name = payload[:task_name]
  error = payload[:error]
  error_message = error[:message]
  error_class = error[:class]
  error_backtrace = error[:backtrace]
rescue => e
  Rails.logger.error(e)
end

# or

class MaintenanceTasksInstrumenter  e
    Rails.logger.error(e)
  end
end

Using Task Callbacks

The Task provides callbacks that hook into its life cycle.

Available callbacks are:

• after_start
• after_pause
• after_interrupt
• after_cancel
• after_complete
• after_error

module Maintenance
  class UpdatePostsTask  { Post.count } do
        Maintenance::UpdatePostsTask.process({
          "title" => "My Title",
          "content" => "Hello World!",
        })
      end

      post = Post.last
      assert_equal "My Title", post.title
      assert_equal "Hello World!", post.content
    end
  end
end

Writing tests for a Task with parameters

Tests for tasks with parameters need to instantiate the task class in order to assign attributes. Once the task instance is setup, you may test #process normally.

# test/tasks/maintenance/update_posts_via_params_task_test.rb

require "test_helper"

module Maintenance
  class UpdatePostsViaParamsTaskTest  { Post.first.content } do
        @task.process(Post.first)
      end
    end
  end
end

Writing tests for a Task that uses a custom enumerator

Tests for tasks that use custom enumerators need to instantiate the task class in order to call #enumerator_builder. Once the task instance is set up, validate that #enumerator_builder returns an enumerator yielding pairs of [item, cursor] as expected.

# test/tasks/maintenance/custom_enumerating_task.rb

require "test_helper"

module Maintenance
  class CustomEnumeratingTaskTest :\ pairs:

```sh-session
bundle exec maintenance_tasks perform Maintenance::ParamsTask \
  --arguments post_ids:1,2,3 content:"Hello, World!"

Running a Task from Ruby

You can also run a Task in Ruby by sending run with a Task name to Runner:

MaintenanceTasks::Runner.run(name: "Maintenance::UpdatePostsTask")

To run a Task that processes CSVs using the Runner, provide a Hash containing an open IO object and a filename to run:

MaintenanceTasks::Runner.run(
  name: "Maintenance::ImportPostsTask",
  csv_file: { io: File.open("path/to/my_csv.csv"), filename: "my_csv.csv" }
)

To run a Task that takes arguments using the Runner, provide a Hash containing the set of arguments ({ parameter_name: argument_value }) to run:

MaintenanceTasks::Runner.run(
  name: "Maintenance::ParamsTask",
  arguments: { post_ids: "1,2,3" }
)

…

ruby
# config/application.rb
module YourApplication
  class Application 
  project: "my-project"
  bucket: "user-data-bucket"

internal:
  service: GCS
  credentials: 
  project: "my-project"
  bucket: "internal-bucket"

# config/initializers/maintenance_tasks.rb

MaintenanceTasks.active_storage_service = :internal

There is no need to configure this option if your application uses only one storage service. Rails.configuration.active_storage.service is used by default.

Customizing the backtrace cleaner

MaintenanceTasks.backtrace_cleaner can be configured to specify a backtrace cleaner to use when a Task errors and the backtrace is cleaned and persisted. An ActiveSupport::BacktraceCleaner should be used.

# config/initializers/maintenance_tasks.rb

cleaner = ActiveSupport::BacktraceCleaner.new
cleaner.add_silencer { |line| line =~ /ignore_this_dir/ }

MaintenanceTasks.backtrace_cleaner = cleaner

If none is specified, the default Rails.backtrace_cleaner will be used to clean backtraces.

Customizing the parent controller for the web UI

MaintenanceTasks.parent_controller can be configured to specify a controller class for all of the web UI engine's controllers to inherit from.

This allows applications with common logic in their ApplicationController (or any other controller) to optionally configure the web UI to inherit that logic with a simple assignment in the initializer.

# config/initializers/maintenance_