#13544 Self-hosting the Kubernetes sandbox plugin: chain of blockers (imageRegistry, runtime image tags, agent-sandbox version, sandbox-cr exec probe)

Summary

Self-hosting the Kubernetes sandbox plugin (@paperclipai/plugin-kubernetes) currently hits a chain of blockers that make it impossible to run an agent end-to-end without patching Paperclip or working around it at the cluster level. I got all the way to a live Sandbox pod with the adapter binary present, and still couldn't complete a run. Filing the whole chain because each piece is independently actionable.

Environment: self-hosted Paperclip v2026.831.1 (latest) on k3s (containerd 2.2.3-k3s1, K8s 1.3x), external PostgreSQL, driver: kubernetes, inCluster: true, adapter claude_local. Tried backend: job then backend: sandbox-cr.

Blocker 1 — `imageRegistry` is unusable (server validation vs plugin usage conflict)

The env config imageRegistry is validated by the server as a URL (zod .url() → { validation: "url", code: "invalid_string" }), but the plugin (rewriteRegistry in image-allowlist.ts) uses it as a bare registry prefix (${registry}/${imageName}${tag}). So:

https://ghcr.io/myorg → passes server validation, but produces the invalid image ref https://ghcr.io/myorg/agent-runtime-claude:v1 → pod InvalidImageName.
ghcr.io/myorg → produces a valid image ref, but the server rejects it: Invalid url on imageRegistry.

There is no value that satisfies both. Additionally, target.imageOverride (the other documented override, gated by imageAllowList) appears not wired through the server — 0 references to imageOverride under server/, so it can never be set for a run. Net: there is no in-app way to point the plugin at a self-hosted runtime image.

Fix: either strip the scheme in rewriteRegistry (and relax the server zod to accept a bare host[:port]/path), or wire target.imageOverride end-to-end.

Blocker 2 — no usable default runtime image tag for self-hosters

adapter-defaults.ts hard-codes ghcr.io/paperclipai/agent-runtime-<adapter>:v1, but the public ghcr.io/paperclipai/agent-runtime-* packages publish only git-<sha> tags — there is no :v1 (that tag appears to exist only in the hosted service's registry). So a self-host install pulls ...:v1 → ImagePullBackOff: not found.

Worse, the latest agent-runtime-images build is git-38d8f37… (2026-08-21), which predates the server release v2026.831.1 (2026-08-31). Combined with Blocker 1, self-hosters have no supported way to get a compatible runtime image.

Fix: publish stable/semver runtime image tags to the public registry in lockstep with server releases, and/or document the exact git-<sha> to use per server version.

Blocker 3 — sandbox-cr requires a specific (older) agent-sandbox version; not documented

The plugin emits apiVersion: agents.x-k8s.io/**v1alpha1** (sandbox-cr-builder.ts), even at HEAD. The current kubernetes-sigs/agent-sandbox release v1.0.2 serves only v1beta1 (v1.0.0 dropped v1alpha1). The newest release that still serves v1alpha1 is v0.5.6. Installing "latest" agent-sandbox silently yields a controller the plugin can't talk to.

Fix: either move the plugin to v1beta1, or pin/document the compatible agent-sandbox version.

Blocker 4 — sandbox-cr adapter setup fails to detect the baked-in binary and times out

With everything above worked around (mirrored image → Kyverno rewrite → agent-sandbox v0.5.6 → backend: sandbox-cr), a Sandbox pod does come up on the correct image, and the adapter binary is present:

$ kubectl exec <sandbox-pod> -- sh -c 'echo $PATH; command -v claude'
PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
/usr/bin/claude          # found instantly

But the run fails:

Timed out while installing the adapter runtime command via:
  if ! command -v 'claude' >/dev/null 2>&1; then ... npm install -g @anthropic-ai/claude-code ... fi
(adapter_failed)   Duration: 8s

So Paperclip's own probe (execution-target.ts: command -v <cmd> via the sandbox exec) does not detect claude, runs the (multi-minute) install, and times out at ~8s. The recovery recheck ("if <detectCommand> is on PATH, continuing") also fails to find it. Since a manual kubectl exec finds the binary instantly, the problem is in the sandbox exec path itself (pod-exec.ts — the @kubernetes/client-node WebSocket exec, whose own comments describe a stdin/EOF race). Effectively the exec-based probe/install is unreliable and/or the setup timeout is far too short for the install fallback it triggers.

Fix: make the adapter-runtime detection robust in the sandbox exec path (or raise/allow-configuring the setup-command timeout, and don't fall into a minutes-long npm install on a fast probe timeout when the binary is already baked into the image).

Net effect

The job backend can't run adapters at all (documented: no multi-command exec). The sandbox-cr backend is reachable only after (1) building/mirroring a runtime image yourself, (2) working around imageRegistry entirely (I used a Kyverno mutating policy to rewrite the image, since no config value works), and (3) pinning agent-sandbox v0.5.6 — and even then Blocker 4 stops the run. Happy to provide more detail / logs on any of these.