Originally published at wostal.eu.
TL;DR: I handed a CI re-platform — GitHub Actions → Argo Workflows, GitHub → Codeberg — to an AI agent.
The rewrite looked correct and passed lint, but it didn't account for one thing: the runtime environment had changed completely.
The old runner was an external VM; the new one is a pod inside the cluster, where Tailscale IPs aren't routable.
Nobody ran the pipeline end-to-end, so every difference stayed as a landmine — six of them.
The worst part wasn't the AI's mistake.
It was that I spent hours blaming the network when the real culprit was a single missing newline in an SSH key.
This is the companion to a different incident.
While I was migrating my k3s control plane from SQLite to etcd, the pipeline that ran that migration turned out to be broken in ways that taught me more than the migration itself.
In this post I'll cover: Where the mess came from — an AI-assisted CI re-platform that nobody validated end-to-end The phantom CNI bug — hours of chasing MTU, Tailscale, firewall, and conntrack The real culprit — a single missing newline in an SSH key Why early failures hide later ones — six latent bugs, surfaced one at a time The lesson about delegating migrations — to a human or an AI Where the mess came from The entire CI had just been migrated from GitHub Actions to Argo Workflows, and from GitHub to Codeberg — and that migration was handed to an AI agent 🙂.
The re-platforming rewrote the logic roughly 1:1, but it didn't catch that the runtime environment had changed fundamentally.
The old runner (GitHub Actions / ARC) behaved one way; the new one is a pod in the cluster — from which Tailscale IPs aren't routable, but private IPs are.
Nobody (human or AI) ran the pipeline end-to-end, so all of those differences stayed as landmines.
That's why there were six latent bugs and the hang I'm about to describe — not random faults, but traces of a GHA→Argo translation that was never validated in the new environment.
The phantom CNI bug After the migration, it turned out the deploy pipeline (CI running in the cluster) had never passed end-to-end — was hanging on .
I spent hours on it and was wrong repeatedly: "It's MTU" → a test with MTU 1240 hung too. "It's the Tailscale overlay" → the private IP hung as well (in the test). "It's the firewall" → I opened the firewall, the public IP hung too. "It's asymmetric routing / conntrack in the CNI" → a pod hung too.
Every one of those tests pointed at the network.
Every one of them was lying to me.
The real culprit Each test was contaminated by a single bug: in the test pods, I mounted the SSH key with from a secret — without a trailing newline.
That produces → the key never loads → behaves randomly (sometimes a hang, sometimes "permission denied").
I was chasing phantoms, because the test rig was broken, not the network.
The breakthrough only came when I did it properly: + on the node, and the key written exactly as CI writes it ().
Then the picture was clean and consistent: from a pod, the private IP (10.0.1.x) works (ssh + sftp + full ), while the Tailscale IP (100.x) hangs — because the pod has no route into the tailnet.
The fix: CI uses the private IP (the firewall doesn't filter the private network anyway).
It worked instantly.
Lesson #1: before you blame the network/CNI/MTU, prove that your test rig isn't broken.
One missing newline in a key cost me hours chasing a non-existent CNI problem.
Reproduce identically to production (here: exactly how CI writes the key), and for "packets vanish midway" use /, not guesses. "Works from my laptop, not from a pod" isn't always the network — sometimes it's a test-rig artifact.
Why early failures hide later ones Here's the second trap.
The deploy pipeline never reached apply+ansible, so each subsequent bug only surfaced once the previous one started working.
There were six in total: missing in the runner image missing (ansible needs it on the control node) a kubeconfig artifact that wasn't passed between step