Pod exec/attach hangs indefinitely in standalone mode with static bearer-token kubeconfig (ReverseProxy Transport not used for WebSocket Upgrade)
Description
Pod exec/attach (the Terminal button) hangs indefinitely with no response when running Headlamp standalone (non-in-cluster) with a kubeconfig whose user: stanza uses a static bearer token: (not a client cert, not insecure-skip-tls-verify). Every other feature (listing/watching nodes, pods, namespaces, events, viewing logs) works correctly against the exact same cluster with the exact same credential.
Environment
- Headlamp
v0.45.0(also reproduced onlatest, same digest) - Deployment: plain
docker runon a jumphost, standalone/non-in-cluster mode, kubeconfig mounted read-only - Cluster: kubeadm-provisioned, control plane v1.33.9, workers v1.32.11
- kubeconfig
user:stanza: statictoken:(ServiceAccount token),certificate-authority-dataset (also reproduced withinsecure-skip-tls-verify: trueinstead — no change)
Steps to reproduce
- Configure Headlamp standalone with a kubeconfig using a static bearer token (not exec-plugin, not client-cert)
- Confirm regular operations work (list nodes/pods,
kubectl auth can-i create pods --subresource=execreturnsyesfor the same identity) - Open a pod's Terminal in the UI, or hit the raw proxy endpoint directly:
curl -v "http://<headlamp>/clusters/<ctx>/api/v1/namespaces/<ns>/pods/<pod>/exec?container=<c>&command=%2Fbin%2Fsh&stdin=true&stdout=true&stderr=true&tty=true" \ -H "Connection: Upgrade" -H "Upgrade: websocket" -H "Sec-WebSocket-Version: 13" -H "Sec-WebSocket-Key: dGhlIHNhbXBsZSBub25jZQ==" - Observe: 0 bytes ever received, connection eventually times out. Browser DevTools shows the WebSocket request as "Finished" with completely empty Response Headers (never even receives
101 Switching Protocols). docker logsshows no exec-specific error at all — onlyhttp: proxy error: context canceledonce the client gives up.- The exact same operation via plain
kubectl exec(using the same kubeconfig, from the same host, over the same network) works instantly.
Suspected root cause
In backend/pkg/kubeconfig/kubeconfig.go, SetupProxy() sets:
proxy.Transport = &userAgentRoundTripper{base: roundTripper, ...}userAgentRoundTripper is a custom wrapping struct, not a raw *http.Transport. Go's net/http/httputil.ReverseProxy has a documented limitation: for Connection: Upgrade requests, its internal upgrade-handling path type-asserts proxy.Transport as *http.Transport to obtain TLSClientConfig/dial settings for a raw hijacked connection, and does not route the upgrade request through Transport.RoundTrip() — the code path where the Authorization Bearer token normally gets attached (via whatever RoundTripper makeTransportFor/rest.TransportFor constructs). Since the assertion fails against a custom wrapper type, the raw hijacked dial falls back to defaults, without the token and without the configured TLS/CA settings.
This matches the reproduction exactly:
- Regular REST calls go through normal
RoundTrip()→ work fine. - The
insecure-skip-tls-verifyvscertificate-authority-dataconfig made zero difference — consistent with the upgrade path never consulting the configured TLS settings at all. - No auth error is ever returned (which you'd expect from a missing/invalid token being rejected quickly) — consistent with a connection that never completes its handshake at all, rather than one that's cleanly rejected.
This may be the same underlying cause behind other open reports of exec/logs silently failing outside the simplest in-cluster/default-transport setups (e.g. #3401), though environments differ (that one is in-cluster + OIDC + oauth2-proxy).
Expected behavior
Pod exec/attach should work with a standalone kubeconfig using a static bearer token, the same way regular REST/watch calls already do.
Possible fix direction
Either:
- Set the Authorization header directly on the outgoing request (e.g. in a
Directorfunction) rather than relying solely onRoundTripper-level injection, so it survives Go's raw-dial Upgrade path, and ensure the transport passed tohttputil.ReverseProxyis (or exposes) a real*http.Transportwith the correctTLSClientConfigso Go's internal type assertion for the hijacked dial succeeds; or - Handle exec/attach WebSocket proxying with an explicit WebSocket-aware proxy implementation instead of relying on
httputil.ReverseProxy's built-in (and limited) upgrade support.
Happy to help test a fix against a real cluster if useful — this was reproduced against a live kubeadm cluster, not just synthetically.
Source: kubernetes-sigs/headlamp