utask

µTask, the Lightweight Automation Engine

µTask is an automation engine built for the cloud. It is:

• simple to operate: only a postgres DB is required
• secure: all data is encrypted, only visible to authorized users
• extensible: you can develop custom actions in golang

µTask allows you to model business processes in a declarative yaml format. Describe a set of inputs and a graph of actions and their inter-dependencies: µTask will asynchronously handle the execution of each action, working its way around transient errors and keeping an encrypted, auditable trace of all intermediary states until completion.

Table of contents

• Real-world examples
• Quick Start
• Operating in production
• Configuration
• Authoring Task Templates
• Extending µTask with plugins
• Contributing
• License

Real-world examples

Here are a few real-world examples that can be implemented with µTask:

Kubernetes ingress TLS certificate provisioning

A new ingress is created on the production kubernetes cluster. A hook triggers a µTask template that:

• generates a private key
• requests a new certificate
• meets the certificate issuer's challenges
• commits the resulting certificate back to the cluster

New team member bootstrap

A new member joins the team. The team leader starts a task specifying the new member's name, that:

• asks the new team member to generate an SSH key pair and copy the public key in a µTask-generated form
• registers the public SSH key centrally
• creates accounts on internal services (code repository, CI/CD, internal PaaS, ...) for the new team member
• triggers another task to spawn a development VM
• sends a welcome email full of GIFs

Payments API asynchronous processing

The payments API receives a request that requires an asynchronous antifraud check. It spawns a task on its companion µTask instance that:

• calls a first risk-assessing API which returns a number
• if the risk is low, the task succeeds immediately
• otherwise it calls a SaaS antifraud solution API which returns a score
• if the score is good, the task succeeds
• if the score is very bad, the task fails
• if it is in between, it triggers a human investigation step where an operator can enter a score in a µTask-generated form
• when it is done, the task sends an event to the payments API to notify of the result

The payments API keeps a reference to the running workflow via its task ID. Operators of the payments API can follow the state of current tasks by requesting the µTask instance directly. Depending on the payments API implementation, it may allow its callers to follow a task's state.

Quick start

Running with docker-compose

Download our latest install script, setup your environment and launch your own local instance of µTask.

mkdir utask && cd utask
wget https://github.com/ovh/utask/releases/latest/download/install-utask.sh
sh install-utask.sh
docker-compose up

All the configuration for the application is found in the environment variables in docker-compose.yaml. You'll see that basic auth is setup for user admin with password 1234. Try logging in with this user on the graphical dashboard: http://localhost:8081/ui/dashboard.

You can also explore the API schema: http://localhost:8081/unsecured/spec.json.

Request a new task:

Get an overview of all tasks:

Get a detailed view of a running task:

Browse available task templates:

Running with your own postgres service

Alternatively, you can clone this repository and build the µTask binary:

make all

Operating in production

The folder you created in the previous step is meant to become a git repo where you version your own task templates and plugins. Re-download and run the latest install script to bump your version of µTask.

You'll deploy your version of µTask by building a docker image based on the official µTask image, which will include your extensions. See the Dockerfile generated during installation.

Architecture

µTask is designed to run a task scheduler and perform the task workloads within a single runtime: work is not delegated to external agents. Multiple instances of the application will coordinate around a single postgres database: each will be able to determine independently which tasks are available. When an instance of µTask decides to execute a task, it will take hold of that task to avoid collisions, then release it at the end of an execution cycle.

A task will keep running as long as its steps are successfully executed. If a task's execution is interrupted before completion, it will become available to be re-collected by one of the active instances of µTask. That means that execution might start in one instance and resume on a different one.

Maintenance procedures

Key rotation

1. Generate a new key with symmecrypt, with the 'storage' label.
2. Add it to your configuration items. The library will take all keys into account and use the latest possible key, falling back to older keys when finding older data.
3. Set your API in maintenance mode (env var or command line arg, see config below): all write actions will be refused when you reboot the API.
4. Reboot API.
5. Make a POST request on the /key-rotate endpoint of the API.
6. All data will be encrypted with the latest key, you can delete older keys.
7. De-activate maintenance mode.
8. Reboot API.

Dependencies

The only dependency for µTask is a Postgres database server. The minimum version for the Postgres database is 9.5

Configuration

Command line args

The µTask binary accepts the following arguments as binary args or env var. All are optional and have a default value:

• init-path: the directory from where initialization plugins (see "Developing plugins") are loaded in *.so form (default: ./init)
• plugins-path: the directory from where action plugins (see "Developing plugins") are loaded in *.so form (default: ./plugins)
• templates-path: the directories where yaml-formatted task templates are loaded from, can be a colon separated list (default: ./templates)
• functions-path: the directory where yaml-formatted functions templates are loaded from (default: ./functions)
• region: an arbitrary identifier, to aggregate a running group of µTask instances (commonly containers), and differentiate them from another group, in a separate region (default: default)
• http-port: the port on which the HTTP API listents (default: 8081)
• debug: a boolean flag to activate verbose logs (default: false)
• maintenance-mode: a boolean to switch API to maintenance mode (default: false)

Config keys and files

Checkout the µTask config keys and files README.

Authentication

The vanilla version of µTask doesn't handle authentication by itself, it is meant to be placed behind a reverse proxy that provides a username through the "x-remote-user" http header. A username found there will be trusted as is, and used for authorization purposes (admin actions, task resolution, etc...).

For development purposes, an optional basic-auth configstore item can be provided to define a mapping of usernames and passwords. This is not meant for use in production.

Extending this basic authentication mechanism is possible by developing an "init" plugin, as described below.

Notification

Every task state change can be notified to a notification backend. µTask implements three differents notification backends: Slack, Opsgenie, and generic webhooks.

Default payload that will be sent for generic webhooks are:

task_state_update notifications:

{
    "message": "string",
    "notification_type": "task_state_update",
    "task_id": "public_task_uuid",
    "title": "task title string",
    "state": "current task state",
    "template": "template_name",
    "requester": "optional",
    "resolver": "optional",
    "steps": "14/20",
    "potential_resolvers": "user1,user2,admin",
    "resolution_id": "optional,public_resolution_uuid",
    "tags": "{\"tag1\":\"value1\"}"
}

task_step_update notifications:

{
    "message": "string",
    "notification_type": "task_step_update",
    "task_id": "public_task_uuid",
    "title": "task title string",
    "state": "current task state",
    "template": "template_name",
    "step_name": "string",
    "step_state": "string",
    "requester": "string",
    "resolver": "string",
    "steps": "14/20",
    "resolution_id": "public_resolution_uuid",
    "tags": "{\"tag1\":\"value1\"}"
}

task_validation notifications:

{
    "message": "string",
    "notification_type": "task_validation",
    "task_id": "public_task_uuid",
    "title": "task title string",
    "state": "TODO",
    "template": "template_name",
    "requester": "optional",
    "potential_resolvers": "user1,user2,admin",
    "tags": "{\"tag1\":\"value1\"}"
}

Notification backends can be configured in the global µTask configuration, as described here.

Authoring Task Templates

Checkout the µTask examples directory.

A process that can be executed by µTask is modelled as a task template: it is written in yaml format and describes a sequence of steps, their interdepencies, and additional conditions and constraints to control the flow of execution.

The user that creates a task is called requester, and the user that executes it is called resolver. Both can be the same user in some scenarios.

A user can be allowed to resolve a task in four ways:

• the user is included in the global configuration's list of admin_usernames
• the user is included in the task's template list of allowed_resolver_usernames
• the user is in a group that is included in the task's template list of allowed_resolver_groups
• the user is included in the task resolver_usernames list

Value Templating

µTask uses the go templating engine in order to introduce dynamic values during a task's execution. As you'll see in the example template below, template handles can be used to access values from different sources. Here's a summary of how you can access values through template handles:

• .input.[INPUT_NAME]: the value of an input provided by the task's requester
• .resolver_input.[INPUT_NAME]: the value of an input provided by the task's resolver
• .step.[STEP_NAME].output.foo: field foo from the output of a named step
• .step.[STEP_NAME].metadata.HTTPStatus: field HTTPStatus from the metadata of a named step
• .step.[STEP_NAME].children: the collection of results from a 'foreach' step
• .step.[STEP_NAME].error: error message from a failed step
• .step.[STEP_NAME].state: current state of the given step
• .step.[STEP_NAME].max_retries: max retries of the given step
• .step.[STEP_NAME].try_count: try count of the given step
• .config.[CONFIG_ITEM].bar: field bar from a config item (configstore, see above)
• .iterator.foo: field foo from the iterator in a loop (see foreach steps below)
• .pre_hook.output.foo: field foo from the output of the step's pre-hook (see pre-hooks)
• .pre_hook.metadata.HTTPStatus: field HTTPStatus from the metadata of the step's pre-hook (see pre-hooks)
• .function_args.[ARG_NAME]: argument that needs to be given in the conifguration section to the function (see functions below)

The following templating functions are available:

| Name | Description