tbot application-tunnel silently stops proxying after proxy pod restart

Expected behavior

When a Teleport proxy pod that a tbot application-tunnel is connected to is terminated (e.g., by Karpenter node disruption, rolling update, or eviction), the tbot should detect the broken connection and re-establish the tunnel through one of the remaining healthy proxy pods. Incoming requests to the tunnel listener should continue to be proxied after a brief reconnection period.

Current behavior

When the proxy pod a tbot is connected to is killed while other healthy proxy pods remain available, the tbot's application-tunnel listener silently stops proxying requests. The tunnel listener continues accepting TCP connections on its configured port but returns timeouts or 504 Gateway Timeout to all requests for minutes before eventually self-healing.

Our guess for the root cause is that tbot's ALPN WebSocket connection upgrade (alpn_conn_upgrade.go:268) has no dial timeout. When a request is routed through a load balancer to a dead proxy backend, the upgrade hangs indefinitely until the middlebox timeout fires (~30s). Multiple sequential requests each hang for 30-47s with zero errors logged, compounding to produce a multi-minute outage.

Draft PR https://github.com/gravitational/teleport/pull/68443 addresses this by adding a DialTimeout to the ALPN dialer.

Root cause analysis

Through extensive reproduction testing (200+ iterations across v18.7.5 and v18.10.0), we identified the failure mechanism:

1. tbot's local_proxy performs a new ALPN WebSocket connection upgrade per downstream request
2. When the proxy pod dies, the K8s EndpointSlice updates within 2 seconds (confirmed by monitoring)
3. However, the load balancer/reverse proxy (Traefik in our case) may still route requests to the dead backend during or after endpoint propagation
4. The ALPN upgrade request hangs silently — upgradeConnThroughWebAPI() performs a raw HTTP request on the connection with no deadline set, so it blocks indefinitely waiting for a response
5. The middlebox eventually times out (30-47s in our environment), but no error is logged by tbot
6. The next request may also hit the dead backend, creating sequential 30-47s hangs
7. Total outage = number of sequential bad-route attempts × middlebox timeout

The CA rotation watcher (carotation/service.go) detects the dead connection via EOF within 1-2 seconds and reconnects immediately. But the application-tunnel's per-request ALPN upgrade path has no equivalent detection or timeout.

Bug details

• Teleport versions tested: 18.7.5 and 18.10.0 (both tbot and proxy) — issue reproduces on both
• tbot config: application-tunnel service type with listen: tcp://0.0.0.0:8080
• Join method: IAM
• Proxy setup: 2 replicas behind a Kubernetes Service (ClusterIP, sessionAffinity: None)
• Platform: Kubernetes (EKS), arm64
• Network path: tbot → NLB (TCP:443) → ALB (TLS termination) → Traefik IngressRoute → K8s Service → proxy pods

Reproduction results

v18.7.5 (100 iterations, kill-one-keep-one):

v18.10.0 (100 iterations, kill-one-keep-one):

EndpointSlice was confirmed to update within 2 seconds in every single iteration across all runs.

Recreation steps

1. Deploy tbot with an application-tunnel service connecting to a Teleport cluster with 2+ proxy replicas behind a load balancer
2. Verify the tunnel is working (HTTP requests through the tunnel reach the target application)
3. Force-delete one proxy pod while leaving at least one other healthy:
   bash复制

   kubectl delete pod <proxy-pod> -n teleport --grace-period=0 --force

4. Wait 45 seconds, then test the tunnel
5. Approximately 10-17% of the time, the tunnel will be stuck for 30s-10min

Debug logs during a 7-minute stall

tbot logs "Performing ALPN WebSocket connection upgrade" then goes completely silent for 30-47s per request. No error is ever logged for the failed upgrade:

22:28:06  ← proxy pod killed
22:28:11 [error] Error occurred whilst watching CA rotations. error=error reading from server: EOF
22:28:12 [info ] Started watching for CA rotations              ← CA watcher recovers in 1s
22:28:13 - 22:29:01  ← connections flowing normally
22:29:01 [debug] Performing ALPN WebSocket connection upgrade.
         ← 42s SILENCE (upgrade hangs on dead backend, no error logged)
22:29:43 [debug] Performing ALPN WebSocket connection upgrade.
         ← 19s SILENCE
22:30:02 [debug] Performing ALPN WebSocket connection upgrade.
22:30:58 [debug] Performing ALPN WebSocket connection upgrade.
         ← 20s SILENCE
22:31:18 [debug] Performing ALPN WebSocket connection upgrade.
         ← 44s SILENCE
22:32:02 [debug] Performing ALPN WebSocket connection upgrade.
22:32:55 [debug] Performing ALPN WebSocket connection upgrade.
         ← 20s SILENCE
22:33:15 [debug] Performing ALPN WebSocket connection upgrade.
         ← 32s SILENCE
22:33:47 [debug] Performing ALPN WebSocket connection upgrade.
         ← 17s SILENCE
22:34:04 [debug] Performing ALPN WebSocket connection upgrade.
22:34:58 [debug] Performing ALPN WebSocket connection upgrade.
         ← 27s SILENCE
22:35:25 [debug] Performing ALPN WebSocket connection upgrade.
         ← 32s SILENCE
22:35:57 [debug] Performing ALPN WebSocket connection upgrade.  ← recovery

10 silent hangs totaling 285s. Zero errors logged by tbot.

ALB access logs confirm the failure point

Time: 2026-07-07T15:35:17Z
Request pending since: 2026-07-07T15:34:47Z (30.002s)
Target: 10.56.80.125:32107 (Traefik pod)
Response: 504 from target (Traefik's 30s proxy timeout to dead backend)
Path: GET /webapi/connectionupgrade

The ALB successfully delivered the request to Traefik, which forwarded it to the dead proxy backend and waited 30s before returning 504. This 504 is never surfaced in tbot's logs.

Related

PR https://github.com/gravitational/teleport/pull/68443 adds DialTimeout to the ALPN dialer and sets a deadline on the raw WebSocket upgrade exchange in upgradeConnThroughWebAPI(). This would cause each failed upgrade to timeout quickly (instead of waiting for the middlebox), allowing the next retry to route to a healthy proxy.

Additionally, exposing application-tunnel health in readiness probes or Prometheus metrics would allow Kubernetes operators to detect and react to stuck tunnels.