Fireworq 是一款轻量级、高性能、不依赖语言的作业队列系统。
Fireworq is a lightweight, high-performance job queue system with the following abilities.
Portability - It is available from ANY programming language which can talk HTTP. It works with a single binary without external dependencies.
Reliability - It is built on top of RDBMS (MySQL), so that jobs won't be lost even if the job queue process dies. You can apply an ordinary replication scheme to the underlying DB for the reliability of the DB itself.
Availability - It supports primary/backup nodes. Only one node becomes primary simultaneously and the others become backup nodes. A backup node will automatically be active when the primary node dies.
Scalability - It always works with a single dispatcher per queue which can concurrently dispatch jobs to workers via HTTP. Scalability of workers themselves should be maintained by a load balancer in the ordinary way. This means that adding a worker will never harm performance of grabbing jobs from a queue.
Flexibility - It supports the following features.
Maintainability - It can be managed on a Web UI. It also provides metrics suitable for monitoring.
Run the following commands and you will get the whole system working all at once. Make sure you have Docker installed before running these commands.
$ docker run -p 8080:8080 fireworq/fireworq --driver=in-memory --queue-default=default
Pressing Ctrl+C will gracefully shut it down.
Note that in-memory driver is not for production use. It is
intended to be used for just playing with Fireworq without a storage
middleware.
First of all, you need a Web server which does an actual work for a job. We call it a 'worker'.
A worker must accept a POST request with a body, which is typically
a JSON value, and respond a JSON result. For example, if you have a
worker at localhost:3000, it must handle a request like the
following.
POST /work HTTP/1.1
Host: localhost:3000
{"id":12345}
HTTP/1.1 200 OK
{"status":"success","message":"It's working!"}
The response JSON must have status field, which describes whether
the job succeeded. It must be one of the following values.
| Value | Meaning |
|---|---|
"success" |
The job succeeded. |
"failure" |
The job failed and it can be retried. |
"permanent-failure" |
The job failed and it cannot be retried. |
Any other values are regarded as "failure". The HTTP status code is
always ignored.
Let's make the job asynchronous using Fireworq. All you have to do is
to make a POST request to Fireworq with a worker URL and a job
payload. If you have a Fireworq docker instance and
your docker host IP (from the container's point of view) is
172.17.0.1, then requesting something like the following will
enqueue exactly the same job in the previous example.
$ curl -XPOST -d '{"url":"http://172.17.0.1:3000/work","payload":{"id":12345}}' http://localhost:8080/job/foo
When Fireworq gets ready to grab this job, it will POST the
payload to the url. When the job is completed on the worker, the
log output of Fireworq should say something like this.
fireworq_1 | {"level":"info","time":1507128673123,"tag":"fireworq.dev","action":"complete","queue":"default","category":"foo","id":2,"status":"completed","created_at":1507128673025,"elapsed":98,"url":"http://172.17.0.1:3000/work","payload":"{\"id\":12345}","next_try":1507128673025,"retry_count":0,"retry_delay":0,"fail_count":0,"timeout":0,"message":"It's working!"}
See the full list of API endpoints for the details of the API.
There is only a set of API endpoints provided by Fireworq itself to inspect running queues. They are useful for machine monitoring but not intended for human use.
Instead, use Fireworqonsole, a powerful Web UI which enables monitoring stats of queues, inspecting running or failed jobs and defining queues and routings.
You can configure Fireworq by providing environment variables on starting a daemon. There are many of them but we only describe important ones here. See the full list for the other variables.
FIREWORQ_MYSQL_DSN
Specifies a data source name for the job queue and the repository
database in a form
user:password@tcp(mysql_host:mysql_port)/database?options.
This is for a manual setup and is mandatory for it.
FIREWORQ_QUEUE_DEFAULT
Specifies the name of a default queue. A job whose category is
not defined via the routing API will be delivered
to this queue. If no default queue name is specified, pushing a job
with an unknown category will fail.
If you already have a queue with the specified name in the job queue database, that one is used. Or otherwise a new queue is created automatically.
FIREWORQ_QUEUE_DEFAULT_POLLING_INTERVAL
Specifies the default interval, in milliseconds, at which Fireworq
checks the arrival of new jobs, used when polling_interval in the
queue API is omitted. The default value is 200.
FIREWORQ_QUEUE_DEFAULT_MAX_WORKERS
Specifies the default maximum number of jobs that are processed
simultaneously in a queue, used when max_workers in the
queue API is omitted. The default value is 20.
暂无开放 Issues，或尚未同步最近议题。