#7649·headlamp

Pod exec/attach hangs indefinitely in standalone mode with static bearer-token kubeconfig (ReverseProxy Transport not used for WebSocket Upgrade)

Author: abhi-bhatraCreated Sep 10, 2026Updated Sep 19, 2026

Description

Pod exec/attach (the Terminal button) hangs indefinitely with no response when running Headlamp standalone (non-in-cluster) with a kubeconfig whose user: stanza uses a static bearer token: (not a client cert, not insecure-skip-tls-verify). Every other feature (listing/watching nodes, pods, namespaces, events, viewing logs) works correctly against the exact same cluster with the exact same credential.

Environment

  • Headlamp v0.45.0 (also reproduced on latest, same digest)
  • Deployment: plain docker run on a jumphost, standalone/non-in-cluster mode, kubeconfig mounted read-only
  • Cluster: kubeadm-provisioned, control plane v1.33.9, workers v1.32.11
  • kubeconfig user: stanza: static token: (ServiceAccount token), certificate-authority-data set (also reproduced with insecure-skip-tls-verify: true instead — no change)

Steps to reproduce

  1. Configure Headlamp standalone with a kubeconfig using a static bearer token (not exec-plugin, not client-cert)
  2. Confirm regular operations work (list nodes/pods, kubectl auth can-i create pods --subresource=exec returns yes for the same identity)
  3. Open a pod's Terminal in the UI, or hit the raw proxy endpoint directly:
    bash
    curl -v "http://<headlamp>/clusters/<ctx>/api/v1/namespaces/<ns>/pods/<pod>/exec?container=<c>&command=%2Fbin%2Fsh&stdin=true&stdout=true&stderr=true&tty=true" \
      -H "Connection: Upgrade" -H "Upgrade: websocket" -H "Sec-WebSocket-Version: 13" -H "Sec-WebSocket-Key: dGhlIHNhbXBsZSBub25jZQ=="
  4. Observe: 0 bytes ever received, connection eventually times out. Browser DevTools shows the WebSocket request as "Finished" with completely empty Response Headers (never even receives 101 Switching Protocols).
  5. docker logs shows no exec-specific error at all — only http: proxy error: context canceled once the client gives up.
  6. The exact same operation via plain kubectl exec (using the same kubeconfig, from the same host, over the same network) works instantly.

Suspected root cause

In backend/pkg/kubeconfig/kubeconfig.go, SetupProxy() sets:

go
proxy.Transport = &userAgentRoundTripper{base: roundTripper, ...}

userAgentRoundTripper is a custom wrapping struct, not a raw *http.Transport. Go's net/http/httputil.ReverseProxy has a documented limitation: for Connection: Upgrade requests, its internal upgrade-handling path type-asserts proxy.Transport as *http.Transport to obtain TLSClientConfig/dial settings for a raw hijacked connection, and does not route the upgrade request through Transport.RoundTrip() — the code path where the Authorization Bearer token normally gets attached (via whatever RoundTripper makeTransportFor/rest.TransportFor constructs). Since the assertion fails against a custom wrapper type, the raw hijacked dial falls back to defaults, without the token and without the configured TLS/CA settings.

This matches the reproduction exactly:

  • Regular REST calls go through normal RoundTrip() → work fine.
  • The insecure-skip-tls-verify vs certificate-authority-data config made zero difference — consistent with the upgrade path never consulting the configured TLS settings at all.
  • No auth error is ever returned (which you'd expect from a missing/invalid token being rejected quickly) — consistent with a connection that never completes its handshake at all, rather than one that's cleanly rejected.

This may be the same underlying cause behind other open reports of exec/logs silently failing outside the simplest in-cluster/default-transport setups (e.g. #3401), though environments differ (that one is in-cluster + OIDC + oauth2-proxy).

Expected behavior

Pod exec/attach should work with a standalone kubeconfig using a static bearer token, the same way regular REST/watch calls already do.

Possible fix direction

Either:

  • Set the Authorization header directly on the outgoing request (e.g. in a Director function) rather than relying solely on RoundTripper-level injection, so it survives Go's raw-dial Upgrade path, and ensure the transport passed to httputil.ReverseProxy is (or exposes) a real *http.Transport with the correct TLSClientConfig so Go's internal type assertion for the hijacked dial succeeds; or
  • Handle exec/attach WebSocket proxying with an explicit WebSocket-aware proxy implementation instead of relying on httputil.ReverseProxy's built-in (and limited) upgrade support.

Happy to help test a fix against a real cluster if useful — this was reproduced against a live kubeadm cluster, not just synthetically.

Source: kubernetes-sigs/headlamp