ort

ort is a Rust interface for performing hardware-accelerated inference & training on machine learning models in the Open Neural Network Exchange (ONNX) format.

Based on the now-inactive onnxruntime-rs crate, ort is primarily a wrapper for Microsoft's ONNX Runtime library, but offers support for other pure-Rust runtimes.

ort with ONNX Runtime is super quick - and it supports almost any hardware accelerator you can think of. Even still, it's light enough to run on your users' devices.

When you need to deploy a PyTorch/TensorFlow/Keras/scikit-learn/PaddlePaddle model either on-device or in the datacenter, ort has you covered.

Documentation

• Guide
• API reference
• Examples
• Migrating from v1.x to v2.0

Support

• Discord: #｜ort-general
• GitHub Discussions

Backers

FOSS projects using ort

Open a PR to add your project here

• Text Embeddings Inference (TEI) uses ort to deliver high-performance ONNX Runtime inference for text embedding models.
• Magika uses ort for neural network-based file type detection.
• retto uses ort for reliable, fast ONNX inference of PaddleOCR models on Desktop and WASM platforms.
• edge-transformers uses ort for accelerated transformer model inference at the edge.
• sbv2-api is a fast implementation of Style-BERT-VITS2 text-to-speech using ort.
• BoquilaHUB uses ort for local AI deployment in biodiversity conservation efforts.
• CamTrap Detector uses ort to detect animals, humans and vehicles in trail camera imagery.
• Ortex uses ort for safe ONNX Runtime bindings in Elixir.
• oar-ocr A comprehensive OCR library, built in Rust with ort for efficient inference.
• FastEmbed-rs uses ort for generating vector embeddings, reranking locally.
• Ahnlich uses ort to power their AI proxy for semantic search applications.
• nicegal is a desktop image gallery app that uses ort for OCR & semantic image search.
• Murmure uses ort as its core engine, leveraging NVIDIA Parakeet to deliver fully local, free, private and cross‑platform Speech‑to‑Text enhanced with LLM post‑processing.
• Valentinus uses ort to provide embedding model inference inside LMDB.
• SilentKeys uses ort for fast, on-device real-time dictation with NVIDIA Parakeet and Silero VAD.
• Xybrid uses ort to run LLMs, ASR, and TTS natively on-device across iOS, Android, Flutter, and Unity apps and games.
• Ultralytics YOLO Rust Inference is a high-performance, pure Rust library and CLI providing fast and efficient interface for running YOLO models using ort.