"Unable to add work while shutting down" during graceful worker shutdown

Describe the bug

When a worker shuts down gracefully, Puma::ThreadPool#<< raises RuntimeError: Unable to add work while shutting down. The error is not handled in Server#process_client, so client_error catches it and the client gets a 500. We expected the connection to just close.

RuntimeError: Unable to add work while shutting down (RuntimeError)
  puma/thread_pool.rb:338:in `block in Puma::ThreadPool#<<'
    raise "Unable to add work while shutting down"
  puma/thread_pool.rb:321:in `synchronize'
  puma/thread_pool.rb:321:in `with_mutex'
  puma/thread_pool.rb:336:in `<<'
  puma/server.rb:531:in `process_client'
    @thread_pool << client

In Server#process_client there are two places that send a keep-alive client back for more work. Only one of them is safe during shutdown (line numbers are from 8.0.2):

if next_request_ready
  # Perf optimization for https://github.com/puma/puma/issues/3788
  if @thread_pool.waiting > 0
    can_loop = true
  else
    @thread_pool << client   # 531: raises if @shutdown is true
    close_socket = false
  end
elsif @queue_requests
  client.set_timeout @persistent_timeout
  if @reactor.add client     # 536: returns false if the reactor is closed
    close_socket = false
  end
end

Reactor#add rescues ClosedQueueError, IOError and returns false, so the caller closes the socket. ThreadPool#<< raises instead. Same situation, but line 531 gives a 500.

I think this is a regression. 6.6.1 and 7.0.0 do not have the @thread_pool << client branch at all. The keep-alive path there only used @reactor.add, which is safe. The branch came in with 7.1.0, from #3794 ("Reintroduce keepalive fast inline behavior", for #3788). We went from 6.6.1 to 8.0.1 and the 500s started right after that.

The branch only runs when @thread_pool.waiting == 0, so you need all threads busy to hit it.

This may hit more people soon

nginx 1.29.7 (March 2026) changed its defaults. keepalive in the upstream block is now on by default, proxy_http_version is 1.1 by default, and nginx no longer sends the Connection header to the upstream. So if you run nginx in front of Puma and you upgrade nginx, you start reusing upstream connections without changing any config, and you can reach this path. That is what happened to us. We run nginx 1.30.4, and the errors started right after we upgraded both nginx and Puma.

Puma config:

workers Integer(ENV.fetch('WEB_CONCURRENCY', '4'))       # 10 on the cluster that hits this
threads min_threads_count, Integer(ENV.fetch('RAILS_MAX_THREADS', '1'))
worker_timeout 240
worker_shutdown_timeout 30
# force_shutdown_after is left at the default (-1)
# enable_keep_alives is left at the default (true)
bind "unix:///run/api3/app.sock"

Command line: bundle exec puma -C config/puma.rb. Cluster mode, behind nginx over a UNIX socket.

We saw it with RAILS_MAX_THREADS=1, and we still see it with 2, but less often. That fits the @thread_pool.waiting > 0 check.

To Reproduce

Sorry, I do not have a reproduction. It only happens with real traffic. So instead I am pointing at the code path, which the stack trace shows directly.

You need all of this at the same time:

1. Cluster mode, and a client that reuses the connection. For us that is nginx with upstream keep-alive, which is the nginx default now.
2. A request finishes, and the next request is already readable on the same connection, so next_request_ready is true.
3. No idle thread right then (@thread_pool.waiting == 0), so the code takes line 531 instead of can_loop = true.
4. The worker gets a graceful SIGTERM at that moment, so the thread pool is already shutting down. A phased restart does this. In our case we also have a small recycler that stops its own worker after N requests.

Expected behavior

A graceful shutdown should not return a 500. Line 531 should do what line 536 does: if the pool is shutting down, stop trying to add the work and close the socket instead of raising.

Desktop (please complete the following information):

• OS: Linux (Ubuntu, Docker)
• Puma Version: 8.0.2
• Ruby Version: 3.4.7

How often

About 106 events in two months, on one internal endpoint, on a service where workers are recycled often.