Agent injection fails with EOF on OpenShift/restricted Kubernetes — preferDownload=false streams 85MB binary over kubectl exec tunnel

What happened?

When running devpod ssh or devpod up against a Kubernetes provider workspace on OpenShift (restricted SCC, STRICT_SECURITY=true), the agent injection loop repeatedly fails with EOF and never establishes a shell connection.

The debug log shows:

info execute inject script
info Received line after pong: ARM-false
info Inject binary
info Attempting to download DevPod agent from: https://github.com/loft-sh/devpod/releases/download/v0.6.15/devpod-linux-amd64
info done exec
info E0827 23:45:02.056128  699285 v2.go:104] "Unhandled Error" err="EOF" logger="UnhandledError"

This repeats indefinitely. The pod is healthy (2/2 Running), SSH works via oc port-forward, and the agent binary is pre-installed at the expected path with a matching version — but injection still runs and fails.

What did you expect to happen?

devpod ssh should connect to the workspace shell. When the agent binary is already present at the expected path with a matching version, injection should be skipped entirely. When injection is needed, it should download the binary inside the container (via curl/wget) rather than streaming 85MB over the kubectl exec tunnel.

Root cause

There are two compounding problems:

1. The version check is post-install verification, not a pre-check skip

After extracting the inject script from the binary, the actual flow is:

# 1. Check if install dir is writable
if (! mkdir -p $INSTALL_DIR 2>/dev/null || ! touch $INSTALL_PATH 2>/dev/null || ...); then
    # 2. If not writable, check sudo → EXIT 1 if sudo needs password
    if ! sudo -nl >/dev/null 2>&1; then
        echo "Error: sudo requires a password..."
        exit 1
    fi
fi

# 3. Remove existing binary
$sh_c "rm -f $INSTALL_PATH 2>/dev/null || true"

# 4. Download or stream new binary
if [ "$PREFER_DOWNLOAD" = "true" ]; then
    download || inject
else
    inject || download
fi

# 5. Post-install version verification (NOT a pre-check skip)
if {{ .ExistsCheck }}; then
    echo "Error: failed to install devpod"
    exit 1
fi

# 6. Execute the actual command
{{ .Command }}

The ExistsCheck is not a "skip injection if version matches" check — it's a post-install verification that runs after the binary has already been removed and re-downloaded.

So even when the agent is pre-installed with a matching version:

1. The script always runs
2. It checks writability of the install dir (fails on OpenShift → sudo check → fails → exit 1)
3. Even if writability passes, it removes the existing binary (rm -f $INSTALL_PATH)
4. Then tries to re-download/stream (fails with EOF on OpenShift)
5. Then checks if the binary exists (it was removed in step 3 and not reinstalled → fails)

There is no pre-check that says "binary already exists with correct version, skip everything."

2. preferDownload=false streams 85MB over the exec tunnel

In pkg/devcontainer/setup.go (line 39), InjectAgent is called with preferDownload=false:

agent.InjectAgent(..., false)  // preferDownload=false

This means the inject script tries to stream the 85MB agent binary over stdin through the kubectl/oc exec tunnel. On OpenShift (and likely any restricted Kubernetes environment), this stream fails with EOF because the exec tunnel has size/time limits that the 85MB binary exceeds.

Meanwhile, pkg/tunnel/container.go correctly sets preferDownload=true for the outer tunnel to the host. The inner inject into the devcontainer is the one that's broken.

The inject script (inject.sh) already has a download() function that uses curl -fsSL inside the container — it just never gets called because PREFER_DOWNLOAD=false forces the inject (stream) path first.

How to reproduce

1. Use the Kubernetes provider on OpenShift with STRICT_SECURITY=true
2. Use a POD_MANIFEST_TEMPLATE that runs as non-root with runAsNonRoot: true and allowPrivilegeEscalation: false
3. Pre-install the DevPod agent binary at the expected path (e.g. /home/vscode/.local/bin/devpod) with a matching version
4. Run devpod ssh <workspace> --debug
5. Observe the inject loop: Inject binary → Attempting to download → done exec → EOF → repeat

Environment

• DevPod Version: v0.6.15
• Operating System: Linux (WSL2)
• Provider: Kubernetes (OpenShift 4.x, restricted SCC)
• STRICT_SECURITY=true
• POD_MANIFEST_TEMPLATE with non-root security context

Suggested fixes

1. Pre-check agent version before running the inject script (most important)

Run the version check as a separate command before the inject script. If the agent already exists at $INSTALL_PATH with the correct version, skip the inject script entirely and proceed directly to {{ .Command }}.

// Before calling inject.InjectAndExecute, check if agent exists with correct version
versionCheck := fmt.Sprintf(`[ "$(%s version 2>/dev/null || echo 'false')" = "%s" ]`, remoteAgentPath, version.GetVersion())
result, err := exec.Command(ctx, versionCheck)
if err == nil && strings.TrimSpace(result) == "" {
    // Agent exists with correct version, skip injection
    return exec.Command(ctx, command)
}
// Otherwise proceed with injection
inject.InjectAndExecute(...)

This would:

• Skip the entire inject script when the agent is already present (common case after first setup)
• Avoid the writability/sudo check entirely
• Avoid removing and re-streaming the 85MB binary
• Make devpod ssh instant when the agent is already installed

2. Default preferDownload=true for container injection

When injecting into a container via a tunnel (kubectl exec / oc exec), prefer downloading the binary inside the container using curl/wget rather than streaming over stdin. The download() function in inject.sh already supports this.

3. Or expose preferDownload as a provider/context option

Let users override the hardcoded false in pkg/devcontainer/setup.go via a context option like AGENT_PREFER_DOWNLOAD=true.

Workaround

We worked around this by:

1. Pre-installing the agent binary in the container image at /home/vscode/.local/bin/devpod
2. Making the directory group-writable by group 0 (OpenShift random UID's group) so the inject script doesn't invoke sudo
3. Adding a fake /usr/local/bin/sudo wrapper that strips sudo flags (OpenShift blocks real sudo via no new privileges)
4. Bypassing devpod ssh entirely with a custom devpod-ssh script that uses oc port-forward + direct SSH to the pod's sshd on port 2222

This works reliably across pod recreation but loses DevPod's SSH integration (git credential forwarding, port forwarding, etc.).

Related issues

• #1934 — Running devpod on kubernetes provider in restricted Pod (non-root, limited capabilities) — same OpenShift/restricted SCC context, different symptom
• #1550 — Waiting for devpod agent to come up, context deadline exceeded — similar inject loop pattern