Airbnb's Open-source Distributed Delayed Job Queueing System
Airbnb's Open-source Distributed Delayed Job Queueing System
Dynein is Airbnb's Open-source Distributed Delayed Job Queueing System. Learn more about Dynein in this blog post: Dynein: Building an Open-source Distributed Delayed Job Queueing System.
Dynein's components are structured as Managed objects in Dropwizard.
We have prepared an example application to demonstrate how to integrate Dynein in your service in ./example.
To run this example, follow the following steps:
Setup localstack.
The following steps will be using the default ports of localstack.
Run this command to create the necessary SQS queues:
aws sqs create-queue --endpoint-url http://localhost:4576 --queue-name example --region us-east-1
aws sqs create-queue --endpoint-url http://localhost:4576 --queue-name inbound --region us-east-1Run this command to create the DynamoDB table:
aws dynamodb create-table --table-name dynein_schedules \
--attribute-definitions AttributeName=shard_id,AttributeType=S AttributeName=date_token,AttributeType=S \
--key-schema AttributeName=shard_id,KeyType=HASH AttributeName=date_token,KeyType=RANGE \
--billing-mode PAY_PER_REQUEST --endpoint-url http://localhost:4569 --region us-east-1cd example && gradle run --args='server example.yml'
In another terminal, use curl to schedule jobs.
The following endpoints are available, which maps to scheduled, immediate, and sqs-delayed jobs:
curl 'localhost:28080/schedule/scheduled'curl 'localhost:28080/schedule/immediate'curl 'localhost:28080/schedule/sqs-delayed'Each endpoint supports optional parameters delaySeconds and name.
For example: curl 'localhost:28080/schedule/scheduled?name=testing-name&delaySeconds=15'
In the original terminal where the example is started, at the scheduled time, the following log lines will be printed:
INFO [2019-12-10 01:59:04,785] com.airbnb.dynein.example.ExampleJob: Running job with name: "testing-name"
INFO [2019-12-10 01:59:04,800] com.airbnb.dynein.example.worker.ManagedWorker: Successfully consumed job from queueHere's an example from ./example/example.yml.
We will use inline comments to explain what each option does.
…No open issues yet, or sync has not completed.