Originally published at wostal.eu.
TL;DR: My Hetzner k3s lab quietly became a platform.
Dozens of operators with leader-election leases hammered the default datastore — SQLite via — until compaction entered a death-spiral: 1.36M rows, a 13.8 GB WAL that wouldn't checkpoint, CPU pinned at 99%, load average 79 on 8 cores.
I stopped the bleeding by truncating the WAL, then migrated the control plane to embedded etcd (7.5 GB SQLite → 313 MB etcd, load 79 → 5).
This is the full postmortem — and the lessons.
This is a war story, not a tutorial.
It's about the moment a homelab stops being a homelab and starts behaving like production — without ever announcing it.
The cluster in question, , is the Hetzner k3s setup I wrote about previously.
It started small.
It did not stay small.
In this post I'll cover: How an overgrown lab broke the default datastore — the kine/SQLite compaction death-spiral The firefight — measuring instead of guessing, and the fix that actually worked The permanent fix — migrating the control plane to embedded etcd, and the honest caveats The meta-lesson — how to recognize when your lab has become a platform A diagnostic runbook — so next time it's minutes, not hours There's a companion piece to this incident.
The CI pipeline that ran this etcd migration was itself freshly — and badly — migrated, and debugging it cost me hours over a single missing newline.
I split that into its own post: I Let an AI Re-Platform My CI Pipeline.
Here's What Broke.
Context: it's "just a homelab" — except it isn't began like any homelab: one k3s node on Hetzner, a few things to play with.
The problem is that over months it quietly became a platform.
A single master node (, 8 vCPU / 16 GB, untainted, and also carrying Longhorn and workloads) now runs: ArgoCD, Kargo, Crossplane/Upbound, CloudNativePG, EMQX, Longhorn, trivy-operator, kubescape, Gatekeeper, Goldilocks/VPA, VictoriaMetrics, Loki, OpenTelemetry, Argo Workflows/Events/Rollouts, kgateway, and more.
Each of those is a solid, production-grade operator.
But all of them together hammer the control plane on k3s's default settings — which means the datastore is SQLite, accessed through , a shim that translates the etcd API into SQL.
This works beautifully… until the lab crosses an invisible threshold and starts behaving like a platform.
At that point you get production-grade failure modes on lab-grade infrastructure.
That's what this post is about.
Part 1 — The firefight: kine's compaction death-spiral The symptom The control plane was pinned: master at 99% CPU, ~42% of it in the kernel (sys), load average climbing 32 → 79 on 8 cores.
The apiserver started spewing: The datastore was so saturated it couldn't even answer a query about its own metrics.
The wrong leads (and the most important lesson: measure, don't guess) My first suspects were "obvious" — and all wrong: Gatekeeper audit ran with every 60s (full live LISTs from the apiserver).
I disabled it and measured: CPU moved by noise.
Goldilocks/VPA — the recommender was writing ~7 checkpoints/s across 142 VPAs.
Disabled it → CPU unchanged.
Lesson #1: verify every "it's definitely X" by scaling it to zero and measuring.
Cutting individual API clients didn't move CPU, because the bottleneck wasn't any client — it was the datastore engine itself.
The actual root cause Only by getting onto the node (SSH over Tailscale — public SSH is firewalled) did the truth show up: Broken down by key: The mechanism: leader-election leases are updated every ~2 seconds by each operator. kine is supposed to delete old revisions (compaction).
Here, ~55,000 dead revisions accumulated per lease (≈31 hours with no compaction) → 1.36M rows → a 7.5 GB database.
This is the classic kine death-spiral: the table grew so large that the compaction query itself began to time out — so compaction never caught up, so the table kept growing.
A feedback loop.
My own remediation from the day before added fuel (a full trivy rescan, deleting 876 reports, a CNPG resyn