k7

k7

Self-hosted secure VM sandboxes for AI compute at scale

k7 aims to make it easy to create, manage and orchestrate lightweight safe VM sandboxes for executing untrusted code, at scale. It is built on battle-tested VM isolation with Kata, Firecracker, QEMU, Longhorn, and Kubernetes — plus k7's own k7d runtime. It is orignally motivated by AI agents that need to run arbitrary code at scale but it is also great for:

• Custom serverless (like AWS Fargate, but yours)
• Hardened CI/CD runners (no Docker-in-Docker risks)
• Blockchain execution layers for AI dApps

100% open‑source (Apache‑2.0). For technical support, write us at: [email protected]

The Tech Stack

k7 is built on:

• Kubernetes for orchestration, with K3s which is prod-ready and a great choice for edge nodes,
• Kata to encapsulate containers into light-weight virtual-machines,
• Firecracker (kfd) for super-fast boots, light footprints and minimal attack surface (with the jailer),
• Devmapper Snapshotter with thin-pool provisioning of logical volumes for efficient disk use across many Firecracker VMs per node,
• QEMU (kql) via Kata when you want a fuller VMM and durable sandbox disks,
• Longhorn for replicated PVC-backed root disks on the QEMU path — named snapshots, restore, disk-only fork, and cross-node mobility,
• k7d — k7's own microVM runtime daemon (Katakate/k7d) with VM-level warm fork (CoW disk+memory) and in-place pause/resume.

Sandbox backends

k7 install --backend provisions one or more backends per node; k7 create --backend … picks one per sandbox. See docs/BACKENDS.md for the architecture and PERFORMANCE.md for the full measurements (Hetzner AX41 node, medians).

* medians of 3 on one Hetzner AX41 node — methodology, ranges, and docker-in-VM numbers are in PERFORMANCE.md. † kfd needs a spare raw disk the lifecycle-bench node didn't have; Show HN measured kfd create→exec 3.74s and fork n/a (rejected). Docker-workload numbers for kfd are in the PERFORMANCE.md Docker benchmark section. ‡ k7-fc VMM pause/resume returns immediately; CRI exec after resume hung on this run (guest_cid=0 retained — CHALLENGES #17). k7d resume→exec is 0.3s.

Also available today

• ️ Docker build / run inside VM sandboxes: k7 create --docker --backend k7d --egress-open builder ubuntu:24.04 (name then image; or --backend k7d-fc). In-guest dockerd, overlay2, forkable. The same --docker flag on Kata (kfd/kql) injects a privileged docker-vehicle with overlay2 on a block disk — kql persists/forks the graph, kfd is ephemeral. --sidecar docker is a deprecated alias. See PERFORMANCE.md
• ⚡ Warm VM fork on the k7d backend: k7 fork CoW-clones a running sandbox's disk and memory in ~5 ms at the VMM; end-to-end through k7/Kubernetes is ~2 s to a Ready pod
• Multi-node clusters (Ansible + Longhorn)
• Cilium CNI with FQDN egress policies (optional Hubble flow observability via k7 install --hubble; off by default, observability only)
• Pause / resume / fork / restore and k7 snapshot lifecycle
• Python SDK: pip install k7-sdk (katakate package deprecated)

See ROADMAP.md for upcoming work (GPU passthrough, …).

Note: k7 is currently in beta and under security review. Use with caution for highly sensitive workloads.

Usage

For usage you need:

• Node(s) that will host the VM sandboxes
• Client from where to send requests

We provide a:

• CLI: to use on the node(s) directly --> apt install k7
• API: deployed automatically by k7 install (toggle with k7 api enable / k7 api disable)
• Python SDK: HTTP client sync/async --> pip install k7-sdk

Current requirements

For the node(s)

• Ubuntu (amd64 or arm64) host.
  • k7d backend is amd64 / x86_64 only (same ISA; Debian calls it amd64, the release tarball is *-x86_64-linux.tar.gz). kfd and kql support amd64 and arm64.
• Hardware virtualization (KVM) available and accessible
  • Check: ls /dev/kvm should exist.
  • This is typically available on your own Linux machine.
  • On cloud providers, it varies.
    • Hetzner (the only one I tested so far) yes for their Robot instances only, i.e. "dedicated": robot.hetzner.com.
    • AWS: only .metal EC2 instances.
    • GCP: virtualization friendly, most instances, with --enable-nested-virtualization flag.
    • Azure: Dv3, Ev3, Dv4, Ev4, Dv5, Ev5 (Intel/AMD x86) or Dpdsv5, Dpldsv5, Epsv5 (ARM64).
    • DigitalOcean: Premium Intel and AMD droplets with nested virtualization enabled.
    • Others: in general, hardware virtualization is not exposed on cloud VPS, so you'll likely want a dedicated / bare metal.
• One raw disk (unformatted, unpartitioned) for the thin-pool that k7 will provision for efficient disk usage of sandboxes.
  • Use ./utils/wipe-disk.sh /your/disk to wipe a disk clean before provisioning. DANGER: destructive - it will remove data/partitions/formatting/SWRAID.
• Ansible (for installer):

  sudo add-apt-repository universe -y
  sudo apt update
  sudo apt install -y ansible
  

• Docker and Docker Compose (for the API):

  curl -fsSL https://get.docker.com | sh
  

Already tested setups:

• Hetzner Robot dedicated with Ubuntu 24.04 and a spare raw NVMe for the kfd thin-pool. Dual-NVMe boxes (no third drive): install the OS on one disk only — see tutorials/k7_hetzner_node_setup.md. (Older PDF that assumed an add-on third NVMe: tutorials/k7_hetzner_node_setup.pdf.)

For the client

Recent Python, or the k7 CLI / k7-sdk from a Linux node or your laptop (API URL + key).

Development on macOS

The .deb / PPA package is Linux-only (amd64/arm64). On a MacBook:

• CLI from source: ./src/k7/cli/dev.sh (same commands as k7; uses uv + PYTHONPATH=src)
• API client from laptop: set K7_API_URL and K7_API_KEY, then dev.sh create / dev.sh list (no --core)
• k7 install targets Linux servers with KVM — run on the node or via SSH, not on macOS locally
• pip install k7-sdk for Python scripts only

Do not install the Ubuntu .deb on macOS.

Quick Start

Get your node(s) ready

The Launchpad PPA currently publishes 0.2.2. For 0.3.1 (HTTPS API, --docker, k7d 0.6.0, HA k7d-fc copy) install the GitHub release .deb, then clone the matching source — k7 install builds k7-api:local from the current working directory:

curl -fsSL -O https://github.com/Katakate/k7/releases/download/v0.3.1/k7_0.3.1_amd64.deb
sudo apt install ./k7_0.3.1_amd64.deb
git clone --branch v0.3.1 https://github.com/Katakate/k7.git
cd k7
sudo apt install -y ansible
curl -fsSL https://get.docker.com | sh

Then let k7 get your node ready:

$ k7 install --backend kfd,kql,k7d
Current task: Reminder about logging out and back in for group changes
  Installing K7 on 1 host(s)... ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 100% 0:01:41
✅ Installation completed successfully!

Optionally pass -v for a verbose output.

Dual-NVMe Hetzner boxes have no third empty disk for the kfd thin-pool. Put Ubuntu on one NVMe (SWRAID 0 / TWO_DISK=1) and leave the other raw — the playbook auto-detects that spare. Do not pin --disk /dev/nvme1n1: NVMe names swap across reboots. Walkthrough: tutorials/k7_hetzner_node_setup.md (this file is also in public Katakate/k7).

The playbook pins k7d 0.6.0. --docker needs the guest docker service (payload on the node); an older k7d fails loudly with this k7d has no docker service; upgrade. Multi-node inventory shapes (2-node server+agent, 3-node --ha) are in src/k7/deploy/inventory.ini.example.

k7 install serves k7-api on NodePort 31007 over HTTPS. The default is a playbook-minted cluster CA (Let's Encrypt cannot issue for a bare IP). Copy /etc/k7/tls/ca.crt off the node and point the CLI at it:

scp root@:/etc/k7/tls/ca.crt ./k7-ca.crt
k7 config set api.url https://:31007
k7 config set api.ca ./k7-ca.crt
k7 config set api.key 

--api-hostname uses Let's Encrypt via a Caddy sidecar (the DNS A record must point at the first master). --api-tls-cert + --api-tls-key installs an operator-supplied pair. --api-insecure-http is today's plain HTTP NodePort and must be called what it is: keys travel in cleartext. None of this is rate limiting; do not write "the API is now secure".

Pass --api-allow-cidr (repeatable, Cilium only) to restrict who can connect. Off by default. Defence in depth for operators who know their client CIDRs — it stacks with TLS and does not replace it. See docs/BACKENDS.md "TLS for k7-api" and "Restricting who can reach k7-api".

Pass --hubble to turn on Cilium Hubble (relay + CLI, no UI) so policy drops are a hubble observe question. Off by default; requires the Cilium CNI (--hubble --cni flannel fails loudly). See docs/BACKENDS.md "Debugging policy drops".

This will install and most importantly connect together the following components (depending on --backend):

• Kubernetes (K3s prod-ready distribution)
• Kata (for container virtualization)
• Firecracker + Jailer + devmapper thin-pool (kfd)
• QEMU via Kata + Longhorn PVC-backed roots (kql)
• k7d daemon + containerd-shim-k7-v1 + RuntimeClass k7 (k7d)
• Optional: Hubble relay + hubble CLI when --hubble is passed

Careful design: config updates will not touch your existing Docker or containerd setups. We chose to use K3s' own containerd for minimal disruption. Installation may however overwrite existing installations of K3s, Kata, Firecracker, Jailer, QEMU/Kata config, or Longhorn.

CLI Usage

You can run workloads directly from the node(s) using the CLI. To create a sandbox, just create a yaml config for it.

k7.yaml example:

…

Running commands

# Create a sandbox (uses k7.yaml in the current directory by default, but you can also pass: -f myfile.yaml)
k7 create

# Or pick a backend explicitly (kfd | kql | k7d — aliases for the full names)
k7 create -f k7.yaml --backend k7d

# List sandboxes
k7 list

# Delete a sandbox
k7 delete my-sandbox-123

# Delete all sandboxes. You can also pass a namespace
k7 delete-all

Fork / pause / snapshot

…

On k7d, the VMM fork itself is ~5