Fix SignalActor concurrency issues and refactor state management based on condition lock
Author: CLFutureXCreated Dec 23, 2025Updated Jul 9, 2026
Description
Problem Background
The current SignalActor uses asyncio.Event to implement mutual exclusion control between trajectory generation and weight update. In the OpenRLHF distributed training scenario, it has the following core issues:
- Mutual Exclusion Logic Failure: Two independent
Eventobjects cannot guarantee operation mutual exclusion. Trajectory generation and weight update may execute simultaneously, leading to model weight dirty writes and abnormal sample generation. - Concurrent Call Risks: Unlocked state modifications may be tampered with by concurrent requests, making the execution results of
set_*methods uncontrollable.
Solution
Refactor SignalActor based on asyncio.Condition and enum states to achieve atomic state management and strict mutual exclusion:
- Introduce State Enum: Define three states (
IDLE/GENERATOR/UPDATE_WEIGHT) to clearly identify the current resource occupant. - Condition Lock Atomic Operations: All state modification and waiting logic are executed in
async with self.condto ensure concurrency safety. - Loop Wait to Prevent Spurious Wakeups: Use
whileloop instead ofifto judge the state, solving the spurious wakeup problem of condition locks. - Compatible with Legacy Interfaces: Retain
set_generating/set_update_weightsmethods, enabling seamless migration without modifying business code.
Source: OpenRLHF/OpenRLHF