Connection slots leak until Threads_connected pins at max_connections, then the listener dies with the process alive

Summary

On dolt 2.2.3, a long-running dolt sql-server under a churn-heavy client workload accumulated connection slots that no socket corresponded to. Threads_connected pinned at exactly max_connections while the OS socket table held roughly 40% of that number. read_timeout_millis did not reclaim the excess. The terminal state was a live server process with no listener bound to its port at all.

We captured the live state before restarting, because the restart erases it. Everything below is measured, not inferred.

The measurement

Threads_connected = 256      (read from the server; 256 was max_connections)
real sockets      ~104       (22 ESTAB + 41 CLOSE-WAIT + 41 FIN-WAIT-2)

About 150 connection slots were held with no socket behind them. read_timeout_millis was 30000. Those slots did not come back.

Once the slot pool is pinned at its ceiling, every new connection goes to the back_log wait queue, which also fills, and the server logs:

level=warning msg="max waiting connections reached. Client rejected. Increase server max_connections and back_log"

That message names back_log, which is the waiting room rather than the exhausted resource. Following it leads to raising a queue depth in front of a pool that has no free slots, which converts fast rejections into slow ones.

Why socket-level observation is misleading here

The socket table read flat at 104 over 45 seconds with CLOSE-WAIT falling. That looks like congestion reaching equilibrium, and it argues against intervening. The sockets were flat; the slots were leaking. The two observables disagreed, and only Threads_connected reflected the resource the error was actually about.

Terminal state

ss -tlnp | grep 29620   -> nothing listening
server pid              -> alive, state S, RSS ~1.9GB
sockets on 29620        -> 0, of any state (down from 104)
dolt.log                -> last line 20:27:35 -04:00, silent afterwards
other dolt servers       -> two other sql-servers on this host still listening

Two unrelated dolt sql-server processes on the same host kept their listeners, so this was not host networking.

Load attribution

We sampled every live client process and walked each to its parent, 251 samples over 15 seconds:

112  diagnostic command, two concurrent runs
117  eight orchestration dispatchers combined
 22  a read-only query with --limit=5000

The dispatchers were not a runaway. Their own traces show 3122 idle sweeps at a 5s backoff ceiling, so about 1.6 sweeps/sec against roughly 38 connections/sec measured. Both diagnostic runs then exited on their own, removing about 45% of arrivals. Availability did not recover. It went to zero and the listener died. Removing load did not release the slots.

Ruled out

Not CPU (threads at ~0% instantaneous, host load 2.20 across 16 cores and falling from 6.54). Not memory. Not file descriptors (1193 of 1048576). Not port drift. Not a second server on the same port.

Client workload shape

This is a multi-agent orchestrator (Gas City). Its CLI opens a short-lived connection per operation and the process frequently exits without a clean COM_QUIT. Connection arrival rate was around 38/sec. That churn pattern is presumably what surfaces this, and it may be what is needed to reproduce it.

The history suggests slow accumulation rather than a sudden event: 629,587 max waiting connections reached lines had accumulated since 2026-07-17, so the server degraded for roughly three weeks before it stopped serving.

Server config

listener:
  port: 29620
  host: 127.0.0.1
  max_connections: 256          # since raised to 512, which only buys time
  back_log: 50
  max_connections_timeout_millis: 5000
  read_timeout_millis: 30000
  write_timeout_millis: 300000
behavior:
  auto_gc_behavior:
    enable: true
    archive_level: 0

dolt version 2.2.3, Linux, ~20 databases under one data_dir.

What would help

1. Slots that outlive their sockets get reclaimed, by read_timeout or otherwise. This is the substantive ask.
2. The rejection message names max_connections and the observed Threads_connected rather than pointing at back_log, since back_log is not the exhausted resource in this state.
3. A server that has lost its listener while the process is still alive is detectable. Right now an external supervisor cannot distinguish it from a healthy server without probing the port, and the process itself looks fine.

We do not have a minimal reproduction. What we have is the live state of a production server at the point of failure, captured deliberately before the restart. Happy to run further diagnostics against the same workload if there is something specific worth measuring.

This looks adjacent to #8932 but distinct: that one is about the variables not being respected, and this is about slots not being released when they are.