⚡ Pure-Rust WebGPU inference engine — OpenAI-API compatible, GGUF native, runs on any GPU. No Python. No llama.cpp. Single binary.
⚡ Pure-Rust WebGPU inference engine — OpenAI-API compatible, GGUF native, runs on any GPU. No Python. No llama.cpp. Single binary.
Shimmy is independently maintained and free forever. Sponsorship funds certification, compatibility work, and releases.
Shimmy will be free forever. No asterisks. No "free for now." No pivot to paid.
Shimmy is a single-binary OpenAI-compatible inference server for GGUF models. Point your existing AI tools at Shimmy and they just work — locally, privately, and free.
Shimmy is the server. Airframe is the engine. Under the hood, Shimmy runs on Airframe (v0.4.0), a pure-Rust WebGPU (WGSL) transformer engine. No C++ toolchain, no Python runtime, no backend flags. 26 models certified across 12 families. Version history: CHANGELOG · Airframe CHANGELOG.
Why this matters:
SHIMMY_MAX_CTX (see Extended Context)12 model families · 26 certified model/quant combinations — every model below passes Shimmy's 3-box certification regimen (MATH + INFERENCE + DETERMINISM) against the certification ledger. Certification applies to the named model/quant combination; architecture recognition does not automatically mean certification. GGUF files load as-is; no recompilation, no hardcoded per-model constants.
Family Model Quants Llama Llama-3.2-1B-Instruct Q4_K_M · Q6_K Llama-3.2-3B-Instruct Q4_K_M Llama-3.1-8B-Instruct Q4_K_M TinyLlama-1.1B-Chat Q4_0 · Q5_K_M · Q6_K Qwen3 Qwen3-0.6B Q4_K_M Qwen3-1.7B Q4_K_M Qwen3-4B Q4_K_M Qwen3-4B-Thinking Q4_K_M Qwen3-8B Q4_K_M Qwen2 Qwen2-0.5B-Instruct Q4_K_M Qwen2-1.5B-Instruct Q4_K_M Qwen2-7B-Instruct Q4_K_M Qwen3.5 Qwen3.5-9B Q4_K_M Phi-3 Phi-3.5-mini-Instruct Q4_K_M Phi-3-mini-4k-Instruct Q4_0 Phi-2 Phi-2 Q4_K_M Gemma-2 Gemma-2-2B-it Q4_K_M Gemma-2-9B-it Q4_K_M (supported; cert: see v2-roadmap) Gemma-4 Gemma-4-12B-coder Q4_K_M Gemma-4-E4B Q4_K_M DeepSeek-R1 DeepSeek-R1-0528-Qwen3-8B Q4_K_M Ministral Ministral-3-14B-Reasoning Q4_K_M StarCoder2 StarCoder2-3B Q4_K_MSafeTensors format (.safetensors) is supported for model loading via safetensors_native. Full Airframe-native inference for SafeTensors remains roadmap work; see docs/v2-roadmap.md.
SHIMMY_MAX_CTX.cargo install shimmy
shimmy serve --model-path /absolute/path/to/model.gguf --bind 127.0.0.1:11435
Then in another terminal:
shimmy list --short
curl -s http://127.0.0.1:11435/v1/chat/completions \
-H 'Content-Type: application/json' \
-d '{"model":"tinyllama-1.1b","messages":[{"role":"user","content":"Say hi in 5 words."}],"max_tokens":32}'
Full install, model acquisition, GPU, VRAM sizing, platform-specific builds: docs/quickstart.md
Shimmy maintains high code quality through comprehensive testing:
# Full test suite (default features = GPU engine)
cargo test --features airframe,huggingface
# Quick CPU-only tests (no GPU required)
cargo test --lib --no-default-features --features huggingface -- --test-threads=1
See docs/ppt-invariant-testing.md for technical details.
** stars and climbing fast** ⏱ <1s startup 100% Rust, no Python
Hacker News · Front Page Again · IPE Newsletter
Companies: Need invoicing? Email [email protected]
Measured on RTX 3060, Shimmy v2.6.0, TinyLlama-1.1B. Your results vary by hardware.
Shimmy is independently maintained. Sponsorship funds certification, compatibility work, and releases.
Current sponsors: ZephyrCloudIO · alistairheath
** Become a Sponsor** · Invoicing
MIT License — see LICENSE. Shimmy will be free forever.
Promise: This will never become a paid product.
Shimmy is infrastructure: it should be invisible. Reliability through comprehensive validation and property-based testing.
Maintainer: Michael A. Kuykendall · Mission: Making local model inference simple and reliable
This project is a safe space. Trans rights are human rights.
If you or someone you love needs support:
No open issues yet, or sync has not completed.