Apache Iggy: Hyper-Efficient Message Streaming at Laser Speed
Apache Iggy: Hyper-Efficient Message Streaming at Laser Speed
Iggy is a persistent message streaming platform written in Rust, supporting QUIC, WebSocket, TCP (custom binary specification) and HTTP (regular REST API) transport protocols, capable of processing millions of messages per second at ultra-low latency.
Iggy provides exceptionally high throughput and performance while utilizing minimal computing resources.
This is not yet another extension running on top of existing infrastructure, such as Kafka or SQL database.
Iggy is a persistent message streaming log built from the ground up using low-level I/O with thread-per-core shared nothing architecture, io_uring and compio for maximum speed and efficiency.
The name is an abbreviation for the Italian Greyhound - small yet extremely fast dogs, the best in their class. See the lovely Fabio & Cookie ❤️
io_uring.io_uring guarantee the best possible performance on modern Linux systems.examples/rust/src/message-headers/message-compression).cargo install iggy-cliThis is the high-level architecture of the Iggy message streaming server, where extremely high performance and ultra low and stable tail latencies are the primary goals. The server is designed to handle high throughput and very low latency (sub-millisecond tail latencies), making it suitable for real-time applications. For more details, please refer to the documentation.
The official releases follow the regular semver (0.7.0) or have latest tag applied (apache/iggy:latest).
We do also publish edge/dev/nightly releases (e.g. 0.7.0-edge.1 or apache/iggy:edge), for both, SDKs and the Docker images, which are typically compatible with the latest changes, but are not guaranteed to be stable, and as the name states, are not recommended for production use.
C++ and PHP are work in progress.
The interactive CLI is implemented under core/cli, to provide the best developer experience. This is a great addition to the Web UI, especially for all the developers who prefer using the console tools.
Iggy CLI can be installed with cargo install iggy-cli and then simply accessed by typing iggy in your terminal.
There's a dedicated Web UI for the server, which allows managing the streams, topics, partitions, browsing the messages and so on. This is an ongoing effort to build a comprehensive dashboard for administrative purposes of the Iggy server. Check the Web UI in the /web directory. The docker image for Web UI is available, and can be fetched via docker pull apache/iggy-web-ui.
The highly performant and modular runtime for statically typed, yet dynamically loaded connectors. Ingest the data from the external sources and push it further to the Iggy streams, or fetch the data from the Iggy streams and push it further to the external sources. Create your own Rust plugins by simply implementing either the Source or Sink trait and build custom pipelines for the data processing.
…The Model Context Protocol (MCP) is an open protocol that standardizes how applications provide context to LLMs. The Iggy MCP Server is an implementation of the MCP protocol for the message streaming infrastructure. It can be used to provide context to LLMs in real-time, allowing for more accurate and relevant responses.
The official Apache Iggy images can be found in Docker Hub, simply type docker pull apache/iggy to pull the image.
You can also find the images for all the different tooling such as Connectors, MCP Server etc. at Docker Hub.
Please note that the images tagged as latest are based on the official, stable releases, while the edge ones are updated directly from latest version of the master branch.
You can find the Dockerfile and docker-compose in the root of the repository. To build and start the server, run: docker compose up.
Additionally, you can run the CLI which is available in the running container, by executing: docker exec -it iggy-server /iggy.
Keep in mind that running the container on the OS other than Linux, where the Docker is running in the VM, might result in the performance degradation.
Also, when running the container, make sure to include the additional capabilities, as you can find in docker-compose file:
cap_add:
- SYS_NICE
security_opt:
- seccomp:unconfined
ulimits:
memlock:
soft: -1
hard: -1The default configuration can be found in config.toml file in core/server directory.
The configuration file is loaded from the current working directory, but you can specify the path to the configuration file by setting IGGY_CONFIG_PATH environment variable, for example export IGGY_CONFIG_PATH=core/server/config.toml (or other command depending on OS).
When config file is not found, the default values from embedded config.toml file are used.
Topic creation accepts two independent policies: durability for message acknowledgments and consumer_offset_durability for explicit offset stores and deletes. Both default to replicated. This means VSR quorum commit without waiting for stable storage. persisted also requires recoverable stable-storage copies on the replication quorum. Both policies normally store data on disk. Poll auto-commit remains asynchronous and is not covered by the poll response's completion.
The data directory is configured with path or IGGY_PATH. The layout beneath it is streams/<stream>/topics/<topic>/partitions/<partition>, with fixed directory names.
The HTTP Iggy-Durability header reports replicated or persisted for awaited writes, and none for early dispatch acceptance.
Segment flush thresholds control scheduling, independently of acknowledgment durability.
Rust HTTP callers can use HttpClient::send_messages_with_durability to read the advertised guarantee alongside confirmations.
The CLI exposes --durability persisted and --consumer-offset-durability persisted on topic create. Select either independently. The policy names describe completion guarantees and do not prescribe an I/O syscall.
For the detailed documentation of the configuratio
No open issues yet, or sync has not completed.