Integration tests fail on low-CPU parallel CI (Debian mass-rebuild)

I'm packaging Puma 8.0.2 for Debian. Santiago Vila's mass-rebuild of the archive found something that doesn't show up on a normal single amd64 build, but fails consistently on his rebuild farm.

Santiago Vila's builds: 245 consecutive rebuilds on small AWS EC2 instances (1–2 vCPUs). Logs: https://people.debian.org/~sanvila/build-logs/puma/

Official archive amd64: Passes all 753 tests. So I don't think Puma is broadly broken on Debian, the integration tests just don't hold up when the full suite runs in parallel on a CPU-starved builder.

The builders are r7a.medium (1 vCPU), r7a.large / m7a.large (2 vCPUs).

What's going wrong

The failures fall into a few patterns depending on how many CPUs the builder has.

On 1-vCPU machines, the most common problem is teardown hanging. The test sends TERM to a Puma child and then blocks in Process.wait2 inside Integration#stop_server. The child doesn't exit before our TEST_CASE_TIMEOUT (600s in the Debian package), so you get TimeoutPrepend::TestTookTooLong. This hits the SSL integration tests especially hard, plus test_control_gc_stats_tcp and test_control_gc_stats_unix.

On 2-vCPU machines, the cluster restart tests are the big ones. test_phased_restart_cluster and test_refork_cluster either time out waiting for a log line like PID: N) booted in Xs, phase: 1, or the whole test takes longer than the hardcoded 7-second limit in test_integration_pumactl.rb:84 (we regularly see ~31 seconds).

Systemd plugin tests turn up on both: test_systemd_notify_usr2_hot_restart_single often fails because RELOADING=1 never shows up in the NOTIFY socket buffer before the assertion runs.

A few others fail less often - wrong exit codes, log lines not appearing in time, connection races - but they look like the same underlying problem: slow builders running a heavily parallelized integration suite.

Local reproduction

I reproduced this on the Debian packaging tree with taskset.

1 vCPU (taskset -c 0, full suite, ~21 minutes): SSL integration teardown timeouts on test_dflt and test_ssl_run_with_localhost_authority; test_term_exit_code and test_term_suppress got exit 1 instead of the expected 15/0.

2 vCPU (taskset -c 0,1, ~3 minutes): test_phased_restart_cluster failed with Expected 31.32 to be < 7. I saw the same thing again on a different CPU pair (~30.5s). Some local runs pass - it's flaky - but sanvila's builds never pass.

Running test_phased_restart_cluster by itself on 1 CPU passes in about 6 seconds. The failures need the full parallelized suite (parallelize_me! on most integration test classes).

Complete list of failing tests

34 unique test methods appeared across the logs (528 total failure/error events; a single build can list several). Sorted by how often each showed up:

TestIntegrationPumactl

Test	Type	Builds	Notes
test_control_gc_stats_tcp	error	96	TestTookTooLong in stop_server — worst single test
test_control_gc_stats_unix	error	39	same teardown hang
test_phased_restart_cluster	failure	47	log timeout or total time > 7s
test_refork_cluster	failure	27	same cluster restart machinery

TestIntegrationSSL

Test	Type	Builds	Notes
test_ssl_run_with_pem	error	24	teardown timeout
test_ssl_run	error	17	teardown timeout
test_ssl_run_with_encrypted_key	error	16	teardown timeout
test_ssl_run_with_encrypted_pem	error	15	teardown timeout
test_ssl_run_with_localhost_authority	error	10	teardown timeout
test_verify_client_cert_roundtrip	error	3	teardown timeout
test_verify_client_cert_roundtrip_tls1_2	error	2	teardown timeout

TestIntegrationSSLSession

Test	Type	Builds	Notes
test_off_tls1_3	error	30	teardown timeout
test_dflt	error	28	teardown timeout
test_1000_10_tls1_2	error	23	teardown timeout
test_dflt_tls1_3	error	22	teardown timeout
test_dflt_tls1_2	error	20	teardown timeout
test_1000_tls1_2	error	17	teardown timeout
test__10_tls1_2	error	9	teardown timeout

TestPluginSystemd

Test	Type	Builds	Notes
test_systemd_notify_usr2_hot_restart_single	failure	24	RELOADING=1 missing from NOTIFY buffer
test_systemd_notify_usr2_hot_restart_single	error	5	teardown timeout (same method, different failure mode)
test_systemd_cluster_notify	error	7	teardown timeout
test_systemd_notify	error	2	teardown timeout
test_systemd_watchdog	error	1	teardown timeout

TestIntegrationSingle

Test	Type	Builds	Notes
test_int_refuse	failure	6	exit 1 instead of expected 0
test_after_booted_and_after_stopped	failure	3	exit 1 instead of expected 15
test_usr2_restart	failure	3	timeout waiting for "Restarting" log
test_usr2_restart_restores_environment	failure	3	timeout waiting for "Restarting" log
test_term_exit_code	error	5	teardown timeout
test_term_suppress	error	4	teardown timeout
test_puma_started_log_writing	error	1	teardown timeout
test_write_to_log	error	1	teardown timeout

Others

Test	Type	Builds	Notes
TestRedirectIO#test_sighup_redirects_io_single	error	9	teardown timeout
TestRequestInvalidMultiple#test_http_11_req_oversize_content_length	failure	5	expected connection error not raised
TestPumaServer#test_chunked_keep_alive_two_back_to_back	failure	3	Expected: true Actual: false
TestResponseHeader#test_illegal_character_in_value_when_override_content_length	error	1	ECONNREFUSED to ephemeral port

That's all 34 methods.

What would help

1. Don't run fork/cluster/SSL integration tests in parallel when Etc.nprocessors <= 2, or skip them on constrained CI.
2. The 7-second phased-restart timing check looks too tight for real builders - consider scaling it or removing it.
3. It would be good to understand why stop_server → wait2 hangs when the full suite runs under CPU pressure — that's the main 1-vCPU failure mode.

What we're doing on the Debian side

Until upstream has a fix, we're skipping all 34 methods above (plus a few that were already excluded on Debian builders, and some arch-specific failures on GNU/Hurd and odd CPUs - see our packaging if you're curious). Happy to provide more logs or try patches.