fox

Fox is dual-licensed MIT OR Apache-2.0 and stays that way. There is no paid tier and no plan for one.

Try it in 30 seconds

# Linux x86_64 — picks the Vulkan build when a GPU is present, CPU otherwise
curl -fsSL https://github.com/ferrumox/fox/releases/latest/download/install.sh | sh

macOS and Windows: build from source (below), or run the Linux installer under WSL2. Prebuilt binaries are Linux x86_64 for now.

# Pull a model and start (qwen3.6 is 22 GB; qwen3.5 is 2.7 GB if you want a quicker first run)
fox pull qwen3.6
fox serve

# Ask something (OpenAI-compatible)
curl http://localhost:8080/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{"model":"qwen3.6","messages":[{"role":"user","content":"Hello!"}],"stream":true}'

# If you already use Ollama — just change the port from 11434 to 8080. That's it.

Performance

Fox wraps llama.cpp, so a single request decoding on its own runs the same kernels llama-server runs. There is no room for fox to be dramatically faster at that, and it isn't. Where fox pulls ahead is when requests arrive together and share a prompt.

Radeon 890M, Vulkan, Llama-3.2-1B-Instruct-Q8_0, 1856-token shared system prompt. Both servers built from the same vendored llama.cpp, one running at a time, arms alternated across 3 rounds. All ranges below are disjoint.

Doubling the clients costs fox 24% more time to first token and llama-server 79%.

That last row is not a typo and it is not buried on purpose: on single-turn requests with short prompts, fox is about 4% behind. That workload cannot see any of the work fox does, because there is no prompt worth reusing. If your traffic looks like that, fox will not make it faster.

Reproduce either one:

scripts/ab_shared_prefix.sh    # concurrent burst behind a shared prompt
scripts/ab_bench.sh            # decode-bound throughput

Full methodology, including two ways these benchmarks produced convincing wrong answers before they produced right ones, is in docs/design/rocm-benchmarking-2026-08.md.

Numbers against Ollama are pending re-measurement on current hardware. The figures that used to sit here were from an RTX 4060 with no recorded methodology, and this project's rule is that a before/after claim comes from scripts/ab_bench.sh or it does not get published.

How it works

Sequences remember what they hold. Every sequence keeps the tokens resident in its KV cache, including the tokens it generated. A new request is matched to the sequence sharing the longest prefix with it and skips the prefill for that overlap. In a chat, the second turn does not re-read the first.

Requests can copy a prefix from a live sequence. This is the part other llama.cpp servers do not do. Slot affinity normally reuses an idle sequence, so when eight requests carrying the same system prompt arrive at once, none of them can reuse anything and all eight prefill the same tokens. Fox copies the shared prefix out of a sibling that is already decoding. llama-server cannot: its slot selection skips busy slots in both its similarity pass and its LRU fallback.

A shared prefix is paid for once. Sequences sharing a prefix share the block budget for it instead of each reserving a copy, so the server admits as much concurrency as the hardware actually holds.

Requests do not queue behind each other. Continuous batching decodes concurrent requests in the same pass, so a long generation for one client does not delay a short question from another.

Works with every tool you already use

No code changes needed — just change the base URL to http://localhost:8080.

Python

from openai import OpenAI

client = OpenAI(base_url="http://localhost:8080/v1", api_key="sk-local")

resp = client.chat.completions.create(
    model="qwen3.6",
    messages=[{"role": "user", "content": "Say hi in 5 words."}],
)
print(resp.choices[0].message.content)

Node.js

import OpenAI from "openai";

const openai = new OpenAI({ baseURL: "http://localhost:8080/v1", apiKey: "sk-local" });

const resp = await openai.chat.completions.create({
  model: "qwen3.6",
  messages: [{ role: "user", content: "Say hi in 5 words." }],
});
console.log(resp.choices[0].message?.content);

IDE configuration

VSCode / Cursor

{ "github.copilot.advanced": { "serverUrl": "http://localhost:8080" } }

Continue.dev (~/.continue/config.json)

{
  "models": [{
    "title": "fox (local)",
    "provider": "openai",
    "model": "qwen3.6",
    "apiBase": "http://localhost:8080/v1"
  }]
}

See examples/ for more integration guides.

GPU support

Fox detects CUDA, ROCm, Metal, and Vulkan at runtime — one binary runs on any hardware.

Backends are compiled as shared libraries and loaded at runtime, which is why one binary covers all of them rather than needing a build per vendor.

Auto-detection priority: CUDA → ROCm → Vulkan → Metal → CPU.

Installation

Linux x86_64

curl -fsSL https://github.com/ferrumox/fox/releases/latest/download/install.sh | sh

It detects /dev/dri and installs the Vulkan build when a GPU is present (AMD/Intel iGPUs included) or the CPU build otherwise, verifies the published checksum, and tells you if $PREFIX/bin is not on your PATH. Override with --vulkan, --cpu, --version vX.Y.Z or --prefix ~/.local.

Or take the tarball yourself — two variants per release:

V=0.20.2
curl -LO https://github.com/ferrumox/fox/releases/download/v$V/fox-$V-x86_64-unknown-linux-gnu-vulkan.tar.gz
tar xzf fox-$V-x86_64-unknown-linux-gnu-vulkan.tar.gz     # drop -vulkan for the CPU build

The .so files in the tarball must stay beside the binary: fox is linked with RPATH=$ORIGIN and looks for its backends nowhere else.

macOS and Windows

No prebuilt binaries yet — the release workflow builds Linux x86_64 only. Either run the Linux installer under WSL2, or build from source:

git clone --recurse-submodules https://github.com/ferrumox/fox
cd fox && cargo build --release --bin fox

--recurse-submodules is not optional: llama.cpp is vendored, not a system dependency.

Build from source

git clone --recurse-submodules https://github.com/ferrumox/fox
cd fox
cargo build --release

GPU backend is detected at runtime — no recompilation needed when switching between CPU, CUDA, and Metal.

Docker

docker run -p 8080:8080 \
  -v ~/.cache/ferrumox/models:/root/.cache/ferrumox/models \
  ferrumox/fox serve

# Or with docker compose
docker compose up

Usage

…

API endpoints

Features

Runs any GGUF model: Llama, Mistral, Gemma, Qwen, DeepSeek and the rest.

Two APIs, no code changes. OpenAI-compatible /v1/* and Ollama-compatible /api/* on the same port. Point an existing client at localhost:8080 and it works.

Prompt reuse that survives concurrency. Sequences keep the tokens they hold, including generated ones, and a new request skips the prefill for whatever prefix it shares. Requests arriving together can copy a shared prefix out of a sequence that is still decoding, and they share the block budget for it rather than each reserving a copy.

Continuous batching. Concurrent requests decode in the same pass instead of queueing.

Speculative decoding. N-gram proposal built in, or a draft model via --draft-model.

Multi-model serving with lazy loading and LRU eviction. No model needs naming up front; fox loads it on first request and evicts by --max-models and --keep-alive-secs.

Structured output and function calling. JSON Schema compiled to GBNF, raw GBNF grammars accepted directly, and tool-call parsers for Hermes, Mistral and Llama 3.

Vision via llama.cpp mtmd (--mmproj), embeddings, and reranking.

LoRA adapters loaded at startup and re-scaled at runtime without a restart.

Runs where the memory is. Multi-GPU layer split (--split-mode, --tensor-split, --main-gpu), MoE expert offload to RAM (--moe-cpu), KV cache quantization (f16, q8_0, q4_0), and a host-RAM prompt cache (--cache-ram) for conversations that should stay warm without holding GPU blocks.

Survives real traffic. Closing a connection frees its GPU memory immediately. Context rolling keeps a generation going when the window fills. Decode failures retry by batch bisection instead of dropping the request.

Operable. Prometheus metrics, optional FOX_API_KEY auth, permissive CORS, a config file at ~/.config/ferrumox/config.toml, model aliases, Docker and systemd units.

Configuration

All flags can also be set via environment variable or ~/.config/ferrumox/config.toml.