Backport #75613 (cancel in-flight exchange sink RPCs on fragment cancellation) to branch-4.0

Summary

SinkBuffer can abort a CN process with an uncaught std::bad_weak_ptr when an exchange sink RPC completes after its fragment has been cancelled. The fix for this exists — #75613, "Cancel in-flight exchange sink RPCs on fragment cancellation" — but it was backported to branch-4.1 and branch-3.5-cc only. branch-4.0 did not receive it.

Requesting a 4.0 backport.

Observed crash (CN 4.0.x, shared-data mode)

throws exception: std::bad_weak_ptr, trace:
     @  __wrap___cxa_throw
     @  std::__throw_bad_weak_ptr()
     @  starrocks::pipeline::SinkBuffer::_try_to_send_rpc(starrocks::TUniqueId const&,
          std::function<void ()> const&)::{lambda(ClosureContext const&, string_view)#4}::operator()
     @  starrocks::DisposableClosure<starrocks::PTransmitChunkResult,
          starrocks::pipeline::ClosureContext>::Run()
     @  starrocks::RecoverableClosure<starrocks::PInternalService_RecoverableStub>::Run()
     @  brpc::Controller::EndRPC(brpc::Controller::CompletionInfo const&)
     @  brpc::Controller::RunEndRPC(void*)
     @  bthread::TaskGroup::task_runner(long)
     @  bthread_make_fcontext

terminate called recursively

query_id and fragment_instance_id are both all-zero in the log line, consistent with the query context having already been destroyed when the completion callback ran. The throw happens on a bthread with no handler, so the process terminates.

Trigger

The race requires a fragment to be torn down while its chunk-transmit RPCs are still in flight. In our case the driver was a rise in client-side query cancellation — callers with short timeouts abandoning queries — combined with higher query concurrency.

Concretely: we raised FE admission concurrency (query_queue_concurrency_limit) from 32 to 64 on an otherwise unchanged cluster. Concurrent queries went from a typical 12–27 to a sustained 47–52. The crash signature appeared within minutes and had zero occurrences in the preceding 24 hours at the lower concurrency. Reverting the limit stopped it.

So the exposure scales with (in-flight exchange RPCs × cancellation rate), which is exactly what #75613 addresses.

Why 4.0 looks like a gap rather than a decision

• #73202 "Fix graceful exit crash in SinkBuffer" was backported to 4.0, so the 4.0 line has the graceful shutdown variant fixed.
• #75613 covers the cancellation variant and went to 4.1 and 3.5-cc but not 4.0.

The result is that 4.0 is protected against the shutdown case but not the cancellation case, which is the one that fires under normal operation rather than during a restart.

Scope of the fix

#75613 touches CN code only:

be/src/exec/pipeline/exchange/sink_buffer.cpp   +25
be/src/exec/pipeline/exchange/sink_buffer.h      +8
be/test/exec/pipeline/exchange_sink_operator_test.cpp  +130

33 lines of production code, no FE changes and no RPC/protocol change, which should make a 4.0 backport low risk.

Versions

• CN: 4.0.8 and 4.0.12, shared-data mode
• Latest 4.0 release (4.0.14) does not contain the fix
• 4.1.1 also predates the branch-4.1 backport, so no currently published release appears to carry it; the backport PR is labelled version:4.1.4

Ask

Backport #75613 to branch-4.0. Happy to test a build if that helps.