基于 Apache DataFusion 的可嵌入式流处理引擎
Denormalized is a fast embeddable stream processing engine built on Apache DataFusion. It currently supports kafka as a real-time source and sink, windowed aggregations, and stream joins.
Denormalized is work-in-progress and we are actively seeking design partners. If you have have a specific use-case you'd like to discuss please drop us a line via a github issue or email [email protected].
Here's an example job that aggregates sensor values from a kafka topic:
…Denormalized also has python bindings in the py-denormalized/ folder. Here is the same example using python:
…The python version is available on pypi: pip install denormalized
Details about developing the python bindings can be found in py-denormalized/README.md
docker run --rm -p 9092:9092 --name emit_measuremetns emgeee/kafka_emit_measurements:latestcargo run --example simple_aggregationWe use SlateDB for state backend. Initialize your Job Context with a custom config and a path for SlateDB backend to store state -
let config = Context::default_config().set_bool("denormalized_config.checkpoint", true);
let ctx = Context::with_config(config)?
.with_slatedb_backend(String::from("/tmp/checkpoints/simple-agg/job1"))
.await;The job with automatically recover from state if a previous checkpoint exists.
A more powerful example can be seen in our Kafka ridesharing example
Denormalized is built and maintained by Denormalized in San Francisco.
暂无开放 Issues,或尚未同步最近议题。