uzu

[](LICENSE) [](https://github.com/trymirai/uzu/actions) [](crates/legacy/uzu/bindings/python) [](https://pypi.org/project/uzu/) [](https://pypi.org/project/uzu/) [](crates/legacy/uzu/bindings/typescript) [](https://www.npmjs.com/package/@trymirai/uzu) [](https://www.npmjs.com/package/@trymirai/uzu) [](crates/legacy/uzu/bindings/swift) [](Package.swift) [](Package.swift) [](https://swift.org) # uzu A high-performance inference engine for AI models. It allows you to deploy AI directly in your app with **zero latency**, **full data privacy**, and **no inference costs**. Key features: - Simple, high-level API - Unified model configurations, making it easy to add support for new models - Traceable computations to ensure correctness against the source-of-truth implementation - Utilizes unified memory on Apple devices - [Broad model support](https://trymirai.com/local-models) ## Quick Start Rust
Add the dependency: ```toml [dependencies] uzu = { git = "https://github.com/trymirai/uzu", branch = "main", package = "uzu" } ``` Run the code below: ``` … ``` Python
Add the dependency: ```bash uv add uzu==0.5.27 ``` Run the code below: ``` … ``` Swift
Add the dependency: ```swift dependencies: [ .package(url: "https://github.com/trymirai/uzu.git", from: "0.5.27") ] ``` Run the code below: ``` … ``` TypeScript
Add the dependency: ```bash pnpm add @trymirai/[email protected] ``` Run the code below: ``` … ```
Everything from model downloading to inference configuration is handled automatically. Refer to the [documentation](https://docs.trymirai.com) for details on how to customize each step of the process. ## Examples You can run any example via `cargo tools example` \<**rust** | **python** | **swift** | **typescript**\> \<**chat** | **chat-cloud** | **chat-shared-instance** | **chat-structured-output** | **quick-start** | **tool-calls**\>: ### Chat In this example, we will download a model and get a reply to a specific list of messages: Rust ``` … ``` Python ``` … ``` Swift ``` … ``` TypeScript ``` … ```
Once loaded, the same `ChatSession` can be reused for multiple requests until you drop it. Each model may consume a significant amount of RAM, so it's important to keep only one session loaded at a time. For iOS apps, we recommend adding the [Increased Memory Capability](https://developer.apple.com/documentation/bundleresources/entitlements/com.apple.developer.kernel.increased-memory-limit) entitlement to ensure your app can allocate the required memory. ### Chat with the cloud model In this example, we will get a reply to a specific list of messages from a cloud model: Rust ``` … ``` Python ``` … ``` Swift ``` … ``` TypeScript ``` … ``` ### Chat with shared instance This example shows how to reuse chat instance without reloading model into memory: Rust ``` … ``` Python ``` … ``` Swift ``` … ``` TypeScript ``` … ``` ### Chat with structured output Sometimes you want the generated output to be valid JSON with predefined fields. You can use `Grammar` to manually specify a JSON schema for the response you want to receive: Rust ``` … ``` Python ``` … ``` Swift ``` … ``` TypeScript ``` … ``` ### Tool calls This example shows how to use external tools: Rust ``` … ``` Python ``` … ``` Swift ``` … ``` TypeScript ``` … ``` ## Development `uzu` is a native Rust crate with bindings available for: - `Swift` via [uniffi-rs](https://github.com/mozilla/uniffi-rs) - `Python` via [pyo3](https://github.com/PyO3/pyo3) - `TypeScript` via [napi-rs](https://github.com/napi-rs/napi-rs) It supports: - Backends: - `metal` - `cpu` - Targets: - `aarch64-apple-darwin` - `aarch64-apple-ios` - `aarch64-apple-ios-sim` - `aarch64-pc-windows-msvc` _(in progress)_ - `aarch64-unknown-linux-gnu` _(in progress)_ - `wasm32-wasip1-threads` _(in progress)_ - `x86_64-apple-darwin` - `x86_64-pc-windows-msvc` _(in progress)_ - `x86_64-unknown-linux-gnu` _(in progress)_
For initial setup we recommend running cargo tools setup, which installs all necessary dependencies (rustup, uv, pnpm, Rust targets, Metal toolchain, ...) if not already present.
To unify cross-language development we introduce cargo tools: - Install language specific dependencies: `cargo tools install typescript` - Build: `cargo tools build rust --targets apple` - Test: `cargo tools test python` - Run example: `cargo tools example swift chat` ## Model Format `uzu` uses its own model format. You can export a model yourself with [lalamo](https://github.com/trymirai/lalamo): ```bash git clone https://github.com/trymirai/lalamo.git cd lalamo uv run lalamo list-models uv run lalamo convert meta-llama/Llama-3.2-1B-Instruct ``` ## CLI You can run `uzu` in CLI mode: ```bash cargo run --release -p cli ``` This launches an interactive app where you can browse, download, and interact with models. You can also preselect a model with `--model`, passing its identifier or repository id: ```bash cargo run --release -p cli -- --model trymirai/Qwen3.5-4B-M ``` If the model is not downloaded yet, the CLI starts downloading it automatically. ## Benchmarks To run benchmarks, pass a downloaded model path, a benchmark task file, and an output path: ```bash cargo run --release -p cli -- bench {MODEL_PATH} {TASK_PATH} {OUTPUT_PATH} ``` ## Server You can also run `uzu` as an OpenAI-compatible HTTP server: ```bash cargo run --release -p cli -- server --model trymirai/Qwen3.5-4B-M ``` The model is loaded on startup (and downloaded first if needed). By default the server listens on `127.0.0.1:8000`; override the address with `--host` and `--port`: ```bash cargo run --release -p cli -- server --model trymirai/Qwen3.5-4B-M --host 0.0.0.0 --port 8080 ``` It exposes the following endpoints, available both at the root and under `/v1`: - `POST /v1/chat/completions` — chat completions, with streaming when `"stream": true`. Honors `temperature`, `top_p`, `top_k`, and `max_tokens`. - `GET /v1/models` — lists the loaded model. ```bash curl http://127.0.0.1:8000/v1/chat/completions \ -H "Content-Type: application/json" \ -d '{ "model": "trymirai/Qwen3.5-4B-M", "messages": [{"role": "user", "content": "Hello!"}], "stream": true }' ``` ## Troubleshooting If you experience any problems, please contact us via [Discord](https://discord.com/invite/trymirai) or [email](mailto:[email protected]). ## License This project is licensed under the MIT License. See the [LICENSE](LICENSE) file for details.