SEGV in test_2_class_group_bandwidth_throttler_fair_shares under sanitize build when its BOOST_REQUIRE fails

Summary

tests/unit/io_queue_test.cc's test_2_class_group_bandwidth_throttler_fair_shares can SEGV (under an ASan/UBSan sanitize build) whenever its own timing-sensitive BOOST_REQUIRE_GE/BOOST_REQUIRE_LE checks fail. The test is legitimately flaky under CPU contention (it compares measured bandwidth ratios with only a ~1.25% tolerance), and whenever that flakiness triggers, the resulting Boost test failure escalates into a real crash instead of a clean test failure.

Found during review of PR #3617 ("core: merge io_desc_read_write and queued_io_request into one allocation"). Reproduces identically on baseline master (confirmed at commit 1c05d41f, before #3617's changes, and again at current origin/master tip 169810b9), so this is a pre-existing bug unrelated to that PR.

Root cause

background_drain (a local RAII-looking helper defined at the top of tests/unit/io_queue_test.cc, used by several tests in this file) starts a seastar::async fiber in its constructor that loops on tio.queue.poll_io_queue() / tio.sink.drain(...) until stop() is called:

struct background_drain {
    io_queue_for_tests& tio;
    bool keep_going;
    future<> done;
    ...
    future<> stop() {
        keep_going = false;
        return std::move(done);
    }
};

It has no destructor that joins this fiber. stop() is only invoked explicitly at the end of the happy path of each test. In test_2_class_group_bandwidth_throttler_fair_shares, cleanup (drain.stop().get(), destroy_scheduling_group(...), destroy_scheduling_supergroup(...)) was placed after the BOOST_REQUIRE_LE/BOOST_REQUIRE_GE checks:

BOOST_REQUIRE_LE(float(bw0) / float(bw1), 4.05);
BOOST_REQUIRE_GE(float(bw0) / float(bw1), 3.95);   // <-- can fail under contention
BOOST_REQUIRE_LE(bw0 + bw1, bandwidth + burst + bw_slack);

drain.stop().get();                                 // never reached on failure
destroy_scheduling_group(sg1).get();
destroy_scheduling_group(sg0).get();
destroy_scheduling_supergroup(ssg).get();

When a BOOST_REQUIRE_* fails, Boost.Test throws boost::execution_aborted to unwind out of the test body. This skips drain.stop().get(), so the background_drain's async fiber is left running, holding a reference to tio (io_queue_for_tests&). As the stack unwinds, tio (and the scheduling groups) are destroyed while the orphaned fiber is still scheduled to run again on the reactor and dereferences the now-freed tio.queue / tio.sink. This is a use-after-free that (depending on what heap memory gets reused in the interim) manifests as heap corruption and a SEGV, sometimes far from the actual dangling access.

Reproduction

Built with:

./configure.py --mode=sanitize
ninja -C build/sanitize tests/unit/io_queue_test

The test is only flaky under CPU contention, so a plain sequential loop rarely triggers it. Reliable repro: run many copies in parallel to create contention:

for j in $(seq 1 16); do
  ./build/sanitize/tests/unit/io_queue_test \
    --run_test=test_2_class_group_bandwidth_throttler_fair_shares \
    --random-seed=$j > out_$j.log 2>&1 &
done
wait

Roughly 1 in 15-20 parallel runs on a 16-core machine triggered the crash for us.

Captured crash output (baseline master, commit 1c05d41f)

/tests/unit/io_queue_test.cc(990): fatal error: in "test_2_class_group_bandwidth_throttler_fair_shares": critical check float(bw0) / float(bw1) >= 3.95 has failed [3.92727017 < 3.9500000000000002]
AddressSanitizer:DEADLYSIGNAL
unknown location(0): fatal error: in "test_2_class_group_bandwidth_throttler_fair_shares": boost::execution_aborted
/tests/unit/io_queue_test.cc(990): last checkpoint
=================================================================
==2679709==ERROR: AddressSanitizer: SEGV on unknown address 0x7bfa4fc10658 (pc 0x0000004d0ea6 bp 0x7bfa4fc10650 sp 0x7bfa4fa7fdb0 T3)
==2679709==The signal is caused by a READ memory access.
*** 2 failures are detected in the test module "Master Test Suite"
src/core/reactor_backend.cc:1696:38: runtime error: member call on address 0x7bfa490d93d8 which does not point to an object of type 'kernel_completion'
0x7bfa490d93d8: note: object has a possibly invalid vptr: abs(offset to top) too big

The BOOST_REQUIRE_GE failure at line 990 immediately precedes the SEGV — consistent with the orphaned background_drain fiber corrupting the heap right after tio is torn down mid-unwind.

Fix

Fixed on branch yk/fix-fair-queue-segv (local, not pushed) in this repo:

1. Give background_drain a destructor that stops and joins its fiber if stop() was never called, so any early exit from the test (exception, early return) can't leave it dangling.
2. Reorder test_2_class_group_bandwidth_throttler_fair_shares to run cleanup (drain.stop().get(), destroy_scheduling_group(...), destroy_scheduling_supergroup(...)) before the BOOST_REQUIRE_* checks that can throw, rather than after.

After the fix: 0 SEGVs in 96 runs (16-way parallel x 6 rounds) that previously reproduced the crash at roughly a 1-in-15-20 rate; test failures now surface as ordinary Boost test failures (the ratio-tolerance flakiness itself is a separate, lower-severity issue worth tracking on its own).