dora

[English](README.md) | [简体中文](README.zh-CN.md)

Website | Python API | Rust API | Guide | Discord

dora 1.0 is out. Read the 1.0 release post.

# Dora **Agentic Dataflow-Oriented Robotic Architecture** -- a 100% Rust framework for building real-time robotics and AI applications. [**User Guide**](https://dora-rs.ai/dora/) | [**用户指南 (中文)**](https://dora-rs.ai/dora/zh-CN/) > Built and maintained with **agentic engineering** -- AI agents do the heavy lifting on code generation, reviews, refactoring, and testing; humans set direction and gate every merge. --- ## Table of Contents - [Features](#features) - [Installation](#installation) - [Quick Start](#quick-start) - [CLI Commands](#cli-commands) - [Dataflow Configuration](#dataflow-configuration) - [Architecture](#architecture) - [Language Support](#language-support) - [Examples](#examples) - [Development](#development) - [Contributing](#contributing) - [License](#license) ## Features ### Performance - **10-17x faster than ROS2 Python** -- 100% Rust internals with zero-copy shared memory IPC for messages >4KB, flat latency from 4KB to 4MB payloads - **Zenoh SHM data plane** -- nodes publish directly via [Zenoh](https://zenoh.io/) shared memory, bypassing the daemon for 35% lower latency and 3-10x higher throughput on large payloads; automatic network fallback for cross-machine - **Apache Arrow native** -- columnar memory format end-to-end with zero serialization overhead; optional [Arrow IPC framing](docs/yaml-spec.md) for self-describing wire format; shared across all language bindings - **Non-blocking event loop** -- Zenoh publishes offloaded to a dedicated drain task; control commands respond in <500ms even under high data throughput ### Developer experience - **Single CLI, full lifecycle** -- `dora run` for local dev, `dora up/start` for distributed prod, plus build, logs, monitoring, record/replay all from one tool - **Declarative YAML dataflows** -- define pipelines as directed graphs, connect nodes through typed inputs/outputs, optional [type annotations](docs/types.md) with static validation, override with environment variables - **Multi-language nodes** -- write nodes in Rust, Python, C, or C++ with native APIs (not wrappers); mix languages freely in one dataflow - **[Reusable modules](docs/modules.md)** -- compose sub-graphs as standalone YAML files with typed inputs/outputs, parameters, optional ports, and nested composition (compile-time expansion, zero runtime overhead) - **Hot reload** -- live-reload Python operators without restarting the dataflow - **Programmatic builder** -- construct dataflows in Python code as an alternative to YAML ### Production readiness - **Fault tolerance** -- per-node restart policies (never/on-failure/always), exponential backoff, health monitoring, circuit breakers with configurable input timeouts - **Distributed by default** -- local shared memory between co-located nodes, automatic [Zenoh](https://zenoh.io/) pub-sub for cross-machine communication, SSH-based [cluster management](docs/distributed-deployment.md) with label scheduling, rolling upgrades, and auto-recovery - **Coordinator HA** -- persistent redb-backed state store (default), daemon auto-reconnect with exponential backoff, dataflow records survive coordinator restart (running dataflow reclaim-across-restart is partial, see the open issue tracker) - **Dynamic topology** -- add and remove nodes from running dataflows via CLI (`dora node add/remove/connect/disconnect`) without restarting - **Soft real-time** -- optional `--rt` flag for mlockall + SCHED_FIFO; per-node `cpu_affinity` pinning in YAML; comprehensive [tuning guide](docs/realtime-tuning.md) for memory locking, kernel params, and container deployment - **OpenTelemetry** -- built-in structured logging with rotation/routing, metrics, distributed tracing, and zero-setup trace viewing via CLI ### Debugging and observability - **Record/replay** -- capture dataflow messages to `.drec` files, replay offline at any speed with node substitution for regression testing - **Topic inspection** -- `topic echo` to print live data, `topic hz` TUI for frequency analysis, `topic info` for schema and bandwidth - **Resource monitoring** -- `dora top` TUI showing per-node CPU, memory, queue depth, network I/O, restart count, and health status across all machines; `--once` flag for scriptable JSON snapshots - **Trace inspection** -- `trace list` and `trace view` for viewing coordinator spans without external infrastructure - **Dataflow visualization** -- generate interactive HTML or Mermaid graphs from YAML descriptors ### Ecosystem - **Communication patterns** -- built-in [service (request/reply)](docs/patterns.md#2-service-requestreply), [action (goal/feedback/result)](docs/patterns.md#3-action-goalfeedbackresult), and [streaming (session/segment/chunk)](docs/patterns.md#4-streaming-sessionsegmentchunk) patterns via well-known metadata keys; no daemon or YAML changes required - **ROS2 bridge** -- bidirectional topics, services, and actions over DDS or native `rmw_zenoh_cpp`-compatible Zenoh; QoS mapping; Arrow-native type conversion - **Node Hub (package manager)** -- pull a reusable node into a dataflow with one line -- `hub: dora-yolo@^0.5` -- with cargo-style versioned resolution, reproducible lockfiles (`--locked`), and typed contracts checked at build time; backed by a git-based [public catalog](https://github.com/dora-rs/dora-hub/) of ready-made nodes for cameras, YOLO, LLMs, TTS, and more. See the [Hub guide](guide/src/hub/overview.md) *(unstable)* - **In-process operators** -- lightweight functions that run inside a shared runtime, avoiding per-node process overhead for simple transformations ## Installation ### From crates.io (recommended) ```bash cargo install dora-cli # CLI (dora command) pip install dora-rs # Python node/operator API ``` ### From source ```bash git clone https://github.com/dora-rs/dora.git cd dora cargo build --release -p dora-cli PATH=$PATH:$(pwd)/target/release # Python API (requires maturin >= 1.8: pip install maturin) # Must run from the package directory for dependency resolution cd apis/python/node && maturin develop --uv && cd ../../.. ``` ### Platform installers **macOS / Linux:** ```bash curl --proto '=https' --tlsv1.2 -LsSf \ https://github.com/dora-rs/dora/releases/latest/download/dora-cli-installer.sh | sh ``` **Windows:** ```powershell powershell -ExecutionPolicy ByPass -c "irm https://github.com/dora-rs/dora/releases/latest/download/dora-cli-installer.ps1 | iex" ``` ### Build features | Feature | Description | Default | |---------|-------------|---------| | `tracing` | OpenTelemetry tracing support | Yes | | `metrics` | OpenTelemetry metrics collection | Yes | | `python` | Python operator support (PyO3) | No | | `redb-backend` | Persistent coordinator state (redb) | Yes | ```bash cargo install dora-cli --features redb-backend ``` ## Quick Start ### 1. Run a Python dataflow > **Important:** The PyPI package is **`dora-rs`**, not `dora`. The import name > is `dora` (`from dora import Node`), but `pip install dora` installs an > unrelated package. ```bash cargo install dora-cli # or use install script below pip install dora-rs numpy pyarrow git clone https://github.com/dora-rs/dora.git && cd dora dora run examples/python-dataflow/dataflow.yml ``` This runs a sender -> transformer -> receiver pipeline. Here's what the Python node code looks like: ```python # sender.py -- sends messages and polls for STOP from dora import Node import pyarrow as pa import time node = Node() sent = 0 while sent < 100: event = node.try_recv() if event is not None and event["type"] == "STOP": break node.send_output("message", pa.array([sent])) sent += 1 time.sleep(0.1) ``` ```python # receiver.py -- receives and prints messages from dora import Node node = Node() for event in node: if event["type"] == "INPUT": print(f"Got {event['id']}: {event['value'].to_pylist()}") elif event["type"] == "STOP": break ``` See the [Python Getting Started Guide](docs/python-guide.md) for a full tutorial, or the [Python API Reference](docs/api-python.md) for complete API docs. ### 2. Run a Rust dataflow ```bash cd examples/rust-dataflow dora run dataflow.yml ``` ### 3. Distributed mode (ad-hoc) ```bash # Terminal 1: start coordinator + daemon dora up # Terminal 2: start a dataflow (--debug enables topic inspection) dora start dataflow.yml --attach --debug # Terminal 3: monitor dora list dora logs dora top # Stop or restart dora stop dora restart --name dora down ``` ### 4. Managed cluster ```bash # Bring up a multi-machine cluster from a config file dora cluster up cluster.yml # Start a dataflow across the cluster dora start dataflow.yml --name my-app --attach # Check cluster health dora cluster status # Tear down dora cluster down ``` See the [Distributed Deployment Guide](docs/distributed-deployment.md) for cluster.yml configuration, label scheduling, systemd services, rolling upgrades, and operational runbooks. The network side — one LAN, a VPN mesh, or isolated subnets joined by zenoh routers — is the [Multi-machine Guide](docs/multi-machine.md). ## CLI Commands ### Lifecycle | Command | Description | |---------|-------------| | `dora run ` | Run a dataflow locally (no coordinator/daemon needed) | | `dora up` | Start coordinator and daemon in local mode | | `dora down` | Tear down coordinator and daemon | | `dora build ` | Run build commands from a dataflow descriptor | | `dora start ` | Start a dataflow on a running coordinator | | `dora stop ` | Stop a running dataflow | | `dora restart ` | Restart a running dataflow (stop + re-start) | ### Monitoring | Command | Description | |---------|-------------| | `dora list` | List running dataflows (alias: `ps`) | | `dora clean` | Remove finished and failed dataflows from the coordinator | | `dora logs [--node ]` | Show logs for a dataflow or node | | `dora top` | Real-time resource monitor (TUI); also `dora inspect top` | | `dora topic list` | List topics in a dataflow | | `dora topic hz ` | Measure topic publish frequency (TUI) | | `dora topic echo ` | Print topic messages to stdout | | `dora topic info ` | Show topic type and metadata | | `dora node list` | List nodes in a dataflow | | `dora node info ` | Show detailed node status, inputs, outputs, and metrics | | `dora node add --from-yaml ` | Add a node to a running dataflow | | `dora node remove ` | Remove a node from a running dataflow | | `dora node connect ` | Add a live mapping between nodes | | `dora node disconnect ` | Remove a live mapping between nodes | | `dora node restart ` | Restart a single node within a running dataflow | | `dora node stop ` | Stop a single node within a running dataflow | | `dora topic pub ` | Publish JSON data to a topic | | `dora param list ` | List runtime parameters for a node | | `dora param get ` | Get a runtime parameter value | | `dora param set ` | Set a runtime parameter (JSON value) | | `dora param delete ` | Delete a runtime parameter | | `dora trace list` | List recent traces captured by the coordinator | | `dora trace view ` | View spans for a specific trace (supports prefix matching) | | `dora record ` | R