buun-llama-cpp

buun-llama-cpp

This is a highly experimental fork of llama.cpp. Use at your own discretion.

A research and development fork of llama.cpp, providing unique KV cache codecs, inference techniques, and bleeding edge features.

VBR — Variable Bit-Rate KV Cache

Why pay 3-bit or 4-bit quality for a context length you only sometimes reach? VBR quantizes the KV cache dynamically as your session grows — giving you the highest-quality cache possible at any given depth. It quantizes the cache layer by layer, using our best KV codecs, from the least-sensitive layers to the most-sensitive, and only as far down the ladder as your VRAM and context actually require.

The cache starts at FP16 and stays there until there is real budget pressure; then it degrades one (layer, side) tensor at a time — first to an aggregate 15.75 bits/value, then 15.51, and so on down the ladder (f16 → turbo8 → turbo4 → turbo3_tcq → turbo2_tcq → turbo1_tcq), following a per-model price order measured on KLD panels. For Qwen35 (16 attention layers) that is 160 distinct steps from FP16 down to turbo1_tcq; for Gemma4-31B (60 layers), 600 — the finest-grained quality control we can give you at every moment of a session.

And because VBR always draws from the best codecs on the ladder, you never have to track KV formats again: new codecs and research roll straight into VBR, so the default cache will always use the best available ladder.

Recommended usage

On a dedicated GPU, just run:

llama-server -m model.gguf

VBR is the default cache. Codec selection is automatic: models whose complete KV geometry supports Turbo use a turbo4 quality floor, while BailingMoE3/Ling falls back to the classic F16→Q8_0→Q4_0 ladder. VBR derives a KV VRAM budget from whatever is left after weights and compute, advertises the largest context that fits without going below the selected codec's default floor (capped at the model's training length), and degrades tiers on the fly as context fills. The cache is still FP16 until memory pressure actually requires compression.

For maximum context, explicitly use -ct vbr. That deliberate opt-in opens the selected codec's complete ladder (turbo1_tcq for Turbo, q4_0 for classic) unless you also set --vbr-floor:

llama-server -m model.gguf -ct vbr

Run with -v to watch the VBR degrade #… steps fire.

Choose the two things you know

VBR balances three quantities: context length, KV VRAM, and minimum quality. Usually specify at most two and let VBR solve the third:

Setting all three usually just over-constrains the same calculation. Pick the two requirements you actually know; VBR can then optimize the remaining dimension instead of receiving three potentially contradictory answers.

Settings

Auto selects the classic ladder for BailingMoE3/Ling models whose head geometry is not supported by TurboQuant:

llama-server -m model.gguf

Classic keeps the model's native KV width and uses ordinary Q8_0/Q4_0 codecs; it does not allocate Turbo rotations or interpret Turbo model-price tables. Its generic order is still strictly banded: every movable KV layer reaches Q8_0 before any layer advances to Q4_0. Portable projected prompt artifacts are currently Turbo-only, so classic mode retains live KV but cold-prefills when a live prefix is no longer available instead of restoring that prefix from the host cache.

Requirements: a CUDA or ROCm backend (turbo-typed KV needs the TurboQuant interface; layers whose KV lands on the CPU fall back to q8_0). Flash attention is required and force-enabled. Dynamic mode uses unified KV (forced automatically with -np > 1). Context-shift / self-extend and slot/session save-restore are disabled in dynamic mode (they would snapshot tier-typed KV that can't restore across a degrade — tier-aware save-restore is planned); context checkpoints stay enabled on hybrid models. Generation stops cleanly when the context fills. Models without a baked price order use a generic cross-model order.

TCQ (trellis-coded KV cache)

Standard KV cache quantization treats each value independently. TCQ constrains the quantization indices to follow a 512-state trellis, giving a much larger effective codebook at the same bit rate (with FWHT rotation and context-adaptive norm scaling on top). At the 3-bit setting, the trellis cuts median KL-divergence by ~40% versus scalar quantization while using slightly fewer bits (3.25 vs 3.50 bpv), and lands perplexity on par with an f16 KV cache.

Paper: Closing the Gap: Trellis-Coded Quantization for KV Cache at 2-3 Bits

Quality (Qwen3.6-27B Q6_K, 16K, RTX 3090)

turbo3 is scalar 3-bit, turbo3_tcq the trellis-coded version at fewer bits — the gap is exactly what the trellis buys. pp = prefill t/s on a 16K prompt, tg = decode t/s at 16K depth.

The trellis cuts median KLD ~40% (and edges PPL below f16) at fewer bits. Its cost lands in prefill — the Viterbi re-encode drops prompt processing to ~818 t/s vs ~1015 for scalar turbo3; decode is barely affected on a dedicated GPU.

Speed

The trellis cost is compute, so it depends on hardware. On dedicated GPUs (e.g. RTX 3090) the fused tensor-core decode path keeps generation at essentially vanilla / f16 speed (the tg numbers above). On weaker-compute hardware (e.g. the Strix Halo iGPU) the cost is exposed and the TCQ types can be up to ~40% slower than their scalar counterparts — there, prefer the scalar or higher-bit codecs.

How it works

1. FWHT rotation with random sign flips converts correlated KV vectors into i.i.d. Gaussian entries
2. Viterbi encoding on a 512-state (3-bit) or 256-state (2-bit) right-shift trellis finds the globally optimal codeword assignment
3. O(1) sliding-window decode -- each value decodes via a bit window lookup, no trellis traversal at inference
4. Context-adaptive alpha -- logarithmic norm scaling formula automatically adjusts dequantization scale per context length

Custom codebooks

Trained codebooks are included in codebooks/. The defaults are compiled into the CUDA kernels, but you can override them:

TURBO_TCQ_CB=codebooks/3bit/product_aware_iter080.bin \
TURBO_TCQ_CB2=codebooks/2bit/product_aware_iter090.bin \
./build/bin/llama-server -m model.gguf -ngl 99 -fa \
  -ctk turbo3_tcq -ctv turbo3_tcq

Codebook training scripts are in scripts/tcq_train_*.py.

Codecs

These are the individual KV cache codecs the fork ships. In practice you rarely pick one by hand — dynamic VBR mixes them per layer automatically and is the recommended default; the codecs below are mainly for testing and comparison.

Measured on Qwen3.6-27B Q6_K at 16K context (18 chunks of the wikitext-2 test set) on an RTX 3090. Median KL-divergence is versus an f16 KV cache (0 = identical logits); pp is prefill throughput on a 16K prompt and tg is decode throughput at 16K depth. Ordered by KL-divergence (quality) — lower is better.

Speculative decoding + Vision (--mmproj-gpu-swap)

On VRAM-constrained GPUs, a speculative context and the vision encoder (mmproj) may not fit in VRAM simultaneously. For example, Qwen3.6-27B Q6_K + MTP uses ~22.6 GiB on a 24 GiB RTX 3090, leaving no room for mmproj's ~1.1 GiB GPU footprint.

--mmproj-gpu-swap solves this by keeping mmproj on CPU at startup, then temporarily unloading MTP or an external draft-dflash sidecar when an image arrives, loading mmproj to GPU for fast encoding (~1-2s instead of 30-60s on CPU), and restoring speculation afterward. MTP is recreated from the target model; an external DFlash sidecar is reloaded from its GGUF.

./build/bin/llama-server -m Qwen3.6-27B-Q6_K.gguf \
  --mmproj mmproj.gguf --spec-type draft-mtp \
  --mmproj-gpu-swap -ngl 99

When combined with auto-fit (no -c flag), the server automatically sizes context to leave room for the swap. With a single slot, it also auto-enables --kv-unified to avoid splitting the KV cache into separate streams, which doubles usable per-slot context.

DeepSeek V4 Flash and Qwen3.8 Flash Next — MoE cache

DeepSeek V4 Flash is much larger than a typical consumer-GPU model, but its routed experts can stay in system RAM while the hot expert tensors are cached in spare VRAM. On a multi-GPU machine, use layer splitting: tensor splitting was substantially slower for this CPU-expert workload even with NVLink.

Choose the MoE cache mode

Start with --fit on --moe-cache auto on both single- and multi-GPU hosts. This is the recommended adaptive path for large CPU-expert models. It preserves normal placement when the complete model fits, and otherwise selects canonical CPU experts when the remaining VRAM can form useful cache pools. Explicit tensor overrides, GPU layers, tensor splits, CPU affinity, and thread counts remain authoritative.

Leave the other resource knobs unset for the first run. The defaults now:

• derive a per-device safety reserve from each GPU'