#1164·OpenRLHF

Fix SignalActor concurrency issues and refactor state management based on condition lock

Author: CLFutureXCreated Dec 23, 2025Updated Jul 9, 2026

Description

Problem Background

The current SignalActor uses asyncio.Event to implement mutual exclusion control between trajectory generation and weight update. In the OpenRLHF distributed training scenario, it has the following core issues:

  1. Mutual Exclusion Logic Failure: Two independent Event objects cannot guarantee operation mutual exclusion. Trajectory generation and weight update may execute simultaneously, leading to model weight dirty writes and abnormal sample generation.
  2. Concurrent Call Risks: Unlocked state modifications may be tampered with by concurrent requests, making the execution results of set_* methods uncontrollable.

Solution

Refactor SignalActor based on asyncio.Condition and enum states to achieve atomic state management and strict mutual exclusion:

  1. Introduce State Enum: Define three states (IDLE/GENERATOR/UPDATE_WEIGHT) to clearly identify the current resource occupant.
  2. Condition Lock Atomic Operations: All state modification and waiting logic are executed in async with self.cond to ensure concurrency safety.
  3. Loop Wait to Prevent Spurious Wakeups: Use while loop instead of if to judge the state, solving the spurious wakeup problem of condition locks.
  4. Compatible with Legacy Interfaces: Retain set_generating/set_update_weights methods, enabling seamless migration without modifying business code.