larql

LARQL

The model IS the database. Query neural network weights like a graph database. No GPU required.

LARQL decompiles transformer models into a queryable format called a vindex (vector index), then provides LQL (Lazarus Query Language) to browse, edit, and recompile the model's knowledge.

VINDEX3 — the successor container — is specified and explained at vindex3.org. The site is where the format is taught, explored and held to its evidence (the Record); this repository is its reference implementation. Install the format-native reader from vindex3.org/get-started, and cite the specification via vindex3.org/cite.

…

Table of Contents

• Quick Start
  • Serve it over HTTP + gRPC
  • Run attention locally, FFN on another machine
  • MoE expert sharding — experts on CPU-only remote machines
  • Publish to HuggingFace — full + slices + collections
  • Vindex Factory — recipe-driven builds
  • Pull with slice awareness
  • Query via LQL
  • Research / interpretability tools
• What is a Vindex?
• Architecture
  • larql-vindex
  • larql-kv
  • larql-lql
• LQL Reference
  • Key Statements
• Patches
• Vindexfile
• Model Support
• Realtime speech (MOSS-TTS-Realtime)
• Benchmarks
  • Vindex Operations
  • Inference Engine (Gemma 3 4B, Apple Silicon M3 Max)
  • MoE / grid (Gemma 4 26B A4B, M3 Max)
  • Pure MoE (GPT-OSS-20B Q6_K, M3 Max)
  • Load-bearing environment flags (serving & measurement)
  • Dense remote-FFN (Gemma 4 31B Q4K, M3 Max, localhost)
• Residual Stream Trace
  • Tiered Context (infinite context without KV cache)
• Mechanistic interpretability surface
• Documentation
• Platform Support
• Install
• Building & Testing
  • Cross-engine correctness check
• License

Quick Start

…

larql extract defaults to --level inference (full local forward pass) stored at f16. No flags needed for the common case.

Extract tiers and options

…

extract-index is kept as a backwards-compatible alias of extract.

Serve it over HTTP + gRPC

larql serve gemma3-4b.vindex --port 8080

Grid traffic uses f16 wire format by default (50% bandwidth vs f32). Opt out with LARQL_F16_WIRE_DISABLE=1. Enable i8 symmetric quantised residuals (75% bandwidth, opt-in) with LARQL_I8_WIRE=1. Wire format is negotiated per-request via Accept/Content-Type headers — and the two directions are independent as of 2026-07-24 (ADR-0025): Content-Type declares the inbound residual format (f32/f16/i8 request encodings), Accept the return format, so asymmetric pairs like f16-in/i8-return are first-class. Responses can carry an opt-in server-latency trailer (x-larql-timing: 1), and larql dec-bench drift scores any wire config's bits/char fidelity against an in-run f32 baseline (0.5% gate). Non-grid clients receive f32 unchanged.

WebSocket streaming on WS /v1/stream:

// Token-by-token generation — send:
{"type": "generate", "prompt": "The capital of France is", "max_tokens": 50}
// Receive one frame per token:
{"type": "token", "text": " Paris", "index": 0}
// Final frame:
{"type": "done", "tokens": 1, "latency_ms": 48.2}
// Abort mid-generation:
{"type": "cancel"}

SSE token streaming is also available on POST /v1/chat/completions with "stream": true.

Run attention locally, FFN on another machine

…

Other presets: browse (DESCRIBE/WALK only, no forward pass), router (MoE router weights only), expert-server (MoE expert weights for remote CPU serving — see below), all (full clone). See larql slice --help for the explicit part list.

MoE expert sharding — experts on CPU-only remote machines

For Mixture-of-Experts models (Gemma 4 26B A4B, Mixtral, etc.), the expert bank can be served from CPU-only machines with no GPU and no VRAM. The laptop runs attention and the router (hot path); the expert servers hold the dormant majority as memory-mapped data.

# Carve the client slice (attn + embed + router — 2.1 GB for 26B A4B Q4_K)
larql slice gemma4-26b-a4b.vindex --preset expert-server \
  -o gemma4-26b-a4b.expert-server.vindex

# Two expert servers — experts 0-63 on one machine, 64-127 on another
larql serve gemma4-26b-a4b.vindex --port 8081 --experts 0-63
larql serve gemma4-26b-a4b.vindex --port 8082 --experts 64-127

# Client dispatches expert calls directly
larql run gemma4-26b-a4b.vindex \
  --moe-shards "0-63=http://expert-a:8081,64-127=http://expert-b:8082" \
  "The capital of France is"

The expert-server preset includes everything the server needs to boot and serve POST /v1/expert/batch calls: embeddings, norms, the interleaved Q4K dense FFN, the per-layer expert weights (layers/), tokenizer, and manifest.

Single server (simplest — one machine holds all experts):

larql serve gemma4-26b-a4b.vindex --port 8080
larql run  gemma4-26b-a4b.vindex --moe-shards "0-127=http://server:8080" "..."

2D layer × expert grid. Layer shards can themselves fan out to expert servers, so both axes scale independently:

# Layer shard — runs attention for layers 0-14, delegates experts to CPU tier
larql serve gemma4-26b-a4b.vindex --port 8091 --layers 0-14 \
  --moe-shards "0-63=http://expert-a:8081,64-127=http://expert-b:8082"

# larql-router routes by layer range; client just sends --ffn to the router
larql-router --port 9090 \
  --shards "0-14=http://layer-a:8091,15-29=http://layer-b:8092"

larql run gemma4-26b-a4b.vindex --ffn http://router:9090 "..."

Deploy expert servers to fly.io (CPU-only, no GPU, tested):

# Publish the expert-server slice to HuggingFace first
larql publish gemma4-26b-a4b.expert-server.vindex \
  --repo myorg/gemma-4-26b-a4b-vindex-expert-server --slices none

# Then deploy — start.sh auto-downloads the vindex on first boot
fly deploy --app larql-expert-server --config deploy/fly/fly.toml --remote-only

See deploy/fly/ for the Dockerfile, fly.toml, and startup script. First boot downloads the vindex from HuggingFace to the persistent volume (~2 min on fly's network); subsequent restarts are instant.

Live demo: https://larql-expert-server.fly.dev serves hf://chrishayuk/gemma-4-26b-a4b-it-vindex-expert-server — a real CPU-only expert server on fly.io that you can point --moe-shards at.

3-tier topology (ADR-0008). When laptop RAM matters, split the embedding table out to its own server:

# Attention-only client (no embed, no FFN — ~310 MB on 4B, 10× smaller than `client`)
larql slice gemma3-4b.vindex --preset attn -o gemma3-4b.attn.vindex

# Embed server slice (embed + tokenizer; paired with ADR-0008 embed-server)
larql slice gemma3-4b.vindex --preset embed -o gemma3-4b.embed.vindex

The 3-tier client + embed server + FFN server split unlocks the "laptop in ~1 GB" version of the dense-remote topology for small models. Full rationale in docs/adr/0007-vindex-distribution.md and docs/adr/0008-embed-server.md.

Publish to HuggingFace — full + slices + collections

Every published vindex carries a versioned on-disk contract. The crates/larql-vindex-spec crate defines the v1 manifest schema — hardened provenance (pinned upstream commit + per-shard safetensors digests), closed enums for extract_level / dtype / quant, and a 20 GiB shard cap. Repos stamp library_name: larql in their model card so the Hub filters them at huggingface.co/models?library=larql. The contract lives in crates/larql-vindex-spec/SPEC.md.

larql publish combines slice + hf publish and adds HuggingFace collections: one run uploads six sibling repos and files them into three nested collections (model / family / library) for discovery.

# One command. Six repos (full + client + attn + embed + server + browse).
# Three collections (model / family / library).
larql publish gemma4-31b.vindex --repo chrishayuk/gemma-4-31b-it-vindex

# Preview without touching HF
larql publish gemma4-31b.vindex --repo chrishayuk/gemma-4-31b-it-vindex --dry-run

Skip-if-unchanged. Each upload compares the local SHA256 against the remote lfs.oid. Files that already match skip the transfer. Re-publishing a ~27 GB server slice where nothing changed re-uploads only the manifest — not 27 GB of weights. Override with --force-upload.

Streaming + progress. Uploads stream the file (no 27 GB-into-RAM pre-read) and report live progress via a per-file bar. An interrupted run picks up on the next invocation: completed files skip via SHA, the interrupted file re-uploads.

Flags: --no-full, --slices client,server, --collections model,family, --model-title, --family, --library-title, --slice-repo-template, --force-upload, --dry-run. Requires HF_TOKEN or ~/.huggingface/token.

Vindex Factory — recipe-driven builds

docs/vindex-factory.md turns the above into a reviewable pipeline: a recipe file merged to main is a promise that a specific vindex exists on the Hub, built from a pinned upstream by a pinned larql, verified before it went public. Recipe schema, build_id, structural validation, capability manifest, card generation, the size/cost estimate, and the PREFLIGHT→RELEASE build driver are all built (crates/larql-factory); MIRROR and REGISTER stay external (§7 — a rig worker's job, not this binary's).

…

Pull with slice awareness

larql pull mirrors publish on the download side: pick a specific sibling, pull them all, or pull a whole collection. Each file gets an indicatif progress bar; hf-hub resumes interrupted downloads from the .incomplete partial on the next run.

…

Bounding server RSS. --ffn-only skips the eager gate warmup at startup (55 GB → 5.6 GB on 31B Q4_K). For steady-state bounds, layer each of these on as needed:

larql serve gemma4-31b.vindex --port 8080 --ffn-only \
  --layers 0-19                    \  # hard bound: this shard serves only layers 0-19
  --max-gate-cache-layers 4        \  # LRU cap on decoded f16 gate heap
  --release-mmap-after-request        # madvise(DONTNEED) post-request (Linux strict)

--layers is the reliable hard bound on both Linux and macOS. --release-mmap-after-request is strict on Linux, advisory on Darwin. See docs/adr/0005-ffn-service-memory-bounds.md for the measured ceilings under each combination.

Query via LQL

larql repl
larql lql 'USE "gemma3-4b.vindex"; DESCRIBE "France";'
larql lql 'USE "hf://chrishayuk/gemma-3-4b-it-vindex"; DESCRIBE "France";'

Research / interpretability tools

All under larql dev (weight extraction, QK rank analysis, OV→gate projection, circuit discovery, trajectory tr