python_backend: error-string shm freed before parent reads it in is_ready() readiness check -> allocator corruption -> permanent hang
Description
qa/L0_backend_python/model_readiness/test.sh hangs indefinitely (never times out) on:
test_is_ready_raises_exceptiontest_is_ready_returns_non_boolean
100% reproducible -- these are the only two sub-tests where is_ready() takes the has_exception branch (i.e. allocates an error string). returns_true/returns_false never hang.
Root cause
In src/pb_stub.cc, Stub::ProcessUserModelReadinessRequest, the error string's unique_ptr<PbString> error_string_shm is scoped inside if (has_exception) { ... }. Its destructor runs at that block's closing brace -- releasing the only shared-memory reference (ref count 1->0) and freeing the bytes back to Boost's allocator before the stub even notifies the parent, let alone before the parent Loads the same handle. The parent then reads/writes an already-freed handle, corrupting the pool's rbtree_best_fit free-list. The corruption doesn't surface until the next unrelated deallocate on that pool spins forever inside Boost's allocator while holding the pool's one mutex -- which every subsequent Construct/Load on that model instance also needs, making the hang permanent.
Confirmed via ps -eLf (stub thread pegged at ~98% CPU) + gdb -p <pid> -batch -ex "thread apply all bt" on both the stub (stuck in the rbtree deallocate path) and the parent tritonserver (blocked in IPCMessage::Create waiting on the same pool mutex).
Fix
Widen error_string_shm's scope to the whole function (alongside readiness_message, which already lives that long), so it isn't destructed until after the notify -> read -> ack handshake with the parent completes:
// before (bug): declared inside if(has_exception){...} — freed
// right after creation, before notify/read/ack.
if (has_exception) {
std::unique_ptr<PbString> error_string_shm;
...
}
// after (fix): declared at function scope — freed only after
// the parent has acked.
std::unique_ptr<PbString> error_string_shm;
if (has_exception) {
...
}
... // notify -> parent reads -> ack happens here
// error_string_shm destructs now, safely.Both libtriton_python.so and triton_python_backend_stub need rebuilding since the fix touches pb_stub.cc.
To reproduce
Run qa/L0_backend_python/model_readiness/test.sh; test_is_ready_raises_exception and test_is_ready_returns_non_boolean hang rather than complete.
Environment
Diagnosed on Linux on IBM Z (s390x), containerized -- but this is a plain use-after-free from generic C++ RAII scoping, not s390x-specific, so it should reproduce on any platform where the has_exception branch runs.
Verification
After moving the declaration and rebuilding both the backend .so and stub binary, both sub-tests passed and the pegged-CPU stub thread no longer reappeared. Happy to open a PR with this fix if useful -- full RCA with code exhibits and gdb backtraces available on request.
Source: triton-inference-server/server