#608·chisel

High CPU usage (300%+) caused by `keepAliveLoop` goroutine leak and nil pointer panic in `handleWebsocket`

Author: testnet0Created Jul 16, 2026Updated Aug 29, 2026

Description

When running chisel server in a Docker container, CPU usage spikes to 300%+ after client sessions disconnect. The process consumes excessive CPU even with no active tunnels, and eventually becomes unresponsive.

Environment

  • Chisel server: v1.11.8
  • Chisel client: v1.11.7 (version mismatch observed)
  • Running in Docker container
  • Reverse tunnel enabled (-R)
  • Reverse proxies: R:2333=>22

Observed Behavior

Top output:

PID USER      PR  NI    VIRT    RES    SHR S  %CPU  %MEM     TIME+ COMMAND
3630019 root      20   0 1270408  16980   7048 S 344.0   0.1      8:15 bin

Goroutine stack traces show:

  • Multiple keepAliveLoop goroutines stuck in [runnable] state, spinning on ssh.(*mux).SendRequest
  • Nil pointer dereference panic in handleWebsocket at server_handler.go:94

Logs show:

  • Clients rapidly reconnecting with exponential backoff (sessions #4 through #14 in ~7 minutes)
  • Version mismatch warnings on every connection

Root Cause Analysis

Bug 1: keepAliveLoop goroutine leak (High)

share/tunnel/tunnel.go:178-193 — The keepAliveLoop function has no context.Context parameter and can only exit when sshConn.SendRequest returns an error:

go
func (t *Tunnel) keepAliveLoop(sshConn ssh.Conn) {
    for {
        time.Sleep(t.Config.KeepAlive)  // no context awareness
        _, b, err := sshConn.SendRequest("ping", true, nil)
        if err != nil {
            break
        }
        if len(b) > 0 && !bytes.Equal(b, []byte("pong")) {
            t.Debugf("strange ping response")
            break
        }
    }
    //close ssh connection on abnormal ping
    sshConn.Close()
}

When an SSH connection dies:

  1. The goroutine may be sleeping in time.Sleep and won't detect the failure until the next SendRequest call (delay up to KeepAlive interval)
  2. SendRequest on a closed mux returns immediately without blocking — the goroutine enters a rapid cycle of: sleep → wake → SendRequest(immediate error) → break → Close → exit
  3. Multiple such goroutines from disconnected sessions remain in [runnable] state simultaneously, causing the Go scheduler to consume excessive CPU managing transitions

With rapid client reconnect cycles (observed in logs), new keepAliveLoop goroutines are spawned on each BindSSH call before the previous ones have exited, causing goroutine accumulation.

Bug 2: Nil pointer dereference panic (Critical)

server/server_handler.go:94 — When the SSH connection closes before the client sends a "config" request, the reqs channel is closed. Reading from a closed channel returns the zero value (nil for *ssh.Request):

go
select {
case r = <-reqs:   // r = nil when channel is closed
case <-time.After(10 * time.Second):
    ...
}
if r.Type != "config" {  // nil pointer dereference → panic!

This panic kills the handleWebsocket handler, leaving associated goroutines (including keepAliveLoop) orphaned with no parent context to clean them up, further contributing to the goroutine leak.

Suggested Fixes

Priority Fix Location
Critical Add nil check: if r == nil { return } after receiving from reqs channel server_handler.go:84-88
High Pass context.Context to keepAliveLoop, replace time.Sleep with context-aware select + time.After tunnel.go:178-193
High Ensure session disconnect explicitly cancels all spawned goroutines server_handler.go, tunnel.go

Reproduction

  1. Run chisel server with reverse tunnel: chisel server -R --reverse
  2. Connect a client with version mismatch (1.11.7 client → 1.11.8 server)
  3. Disconnect the client (network interruption or manual stop)
  4. Observe CPU usage climb as keepAliveLoop goroutines accumulate
  5. Reconnect repeatedly to accelerate the issue