[Core] Worker hangs at exit and leaks as ray::IDLE when log rotation is on and a child process holds the redirected stream

What happened + What you expected to happen

We run KubeRay batch jobs with fractional GPU tasks (num_gpus=0.2). GPU tasks default to max_calls=1, so each worker handles one task and exits.

The jobs use RAY_ROTATION_MAX_BYTES=100MiB.

After running for a few hours, exited workers start piling up on the nodes. Ray reports them as exited normally:

is_alive: false
exit_type: INTENDED_USER_EXIT
Worker exits with an exit code 0. Exited because worker reached max_calls=1 for this method.

The Linux processes are still alive as ray::IDLE, though, each holding about 3 GB RSS because they import torch and deepspeed. Eventually the node runs out of memory. We hit this in two long-running batch jobs.

We traced the hang to worker log rotation.

If a task spawns a child process, that child inherits stdout/stderr, including the log rotation pipe. If the child outlives the worker, it keeps the pipe write end open. Worker shutdown then waits for the pipe reader to see EOF, which never arrives.

Killing the child that holds the pipe makes the stuck worker exit within about 0.1s.

On a stuck worker running 2.44.0:

• main thread: Py_Exit -> exit handler -> SyncOnStreamRedirection, blocked on a futex
• PipeReaderThd: blocked in read(), waiting for EOF
• the log rotation pipe still has a write end open in the process tree

We see the same fd state in production and in the repro.

With RAY_ROTATION_MAX_BYTES > 0, non-driver workers redirect stdout/stderr through pipes.

RedirectStream dup2s the pipe write end onto fd 1/2, so child processes inherit it as stdout/stderr. subprocess(..., close_fds=True) does not help here because fd 1 and 2 are still inherited.

During shutdown, Ray tears down the redirection handles:

• 2.44.0: SyncOnStreamRedirection via std::atexit in stream_redirection_utils.cc
• current master: destruction of the static redirection_file_handles map in stream_redirection.cc

Both paths eventually wait on:

promise->get_future().get();

in pipe_logger.cc, with no timeout.

If a child still holds the write end, that wait does not complete.

Reproduces on:

• 2.44.0
• 2.49.2
• 2.58.0

Does not reproduce on 2.40.0 through 2.43.0.

The rotation machinery existed before 2.44.0, but worker stdout/stderr started using it in 2.44.0 in core_worker_process.cc.

• #49563 added worker log rotation
• #50248 fixed a similar shutdown hang where the worker's own fds kept the pipe open

The remaining case is a child process holding the write end.

Workaround

We currently run with:

RAY_ROTATION_MAX_BYTES=0

That uses the direct-file path and avoids the pipe.

RAY_LOG_TO_STDERR=1 also avoids this path.

Expected behavior

Worker shutdown should not block forever on a child process that still holds the log pipe.

We have a patch that replaces the unbounded wait with std::future::wait_for, adds a RAY_ROTATION_DRAIN_TIMEOUT_MS knob, and includes a regression test. We can open a PR.

Versions / Dependencies

• Ray: 2.44.0 (production, where we hit it), 2.49.2, 2.58.0: all reproduce. 2.40.0, 2.41.0, 2.42.0, 2.43.0: do not.
• Python: 3.12 in the version matrix, also reproduced with 3.14 on 2.58.0. The wait is in C++, so the Python version should not matter.
• OS: reproduced on macOS 26.5 with the script below; the production hit is Linux on KubeRay. The redirect code is shared between the two.
• No GPU needed for the repro. In production the tasks held num_gpus=0.2, which forces max_calls=1; the repro sets max_calls=1 directly so it runs on CPU.

Reproduction script

The env var must be in the worker's environment, so export it before starting the script (setting os.environ inside the task is too late):

import os
import subprocess
import time

import ray

@ray.remote(max_calls=1)
def task():
    # A child that outlives the task. With rotation enabled, fd 1/2 of the
    # worker are the log rotation pipe, and this child inherits them.
    subprocess.Popen(["/bin/sleep", "600"])
    return os.getpid()

ray.init(num_cpus=1, include_dashboard=False, log_to_driver=False)
worker_pid = ray.get(task.remote())
time.sleep(5)

try:
    os.kill(worker_pid, 0)
    print(f"worker {worker_pid} is stuck alive after its task: ray::IDLE ghost")
except ProcessLookupError:
    print("worker exited normally")

RAY_ROTATION_MAX_BYTES=104857600 python repro.py   # worker hangs, leaks as ray::IDLE
RAY_ROTATION_MAX_BYTES=0 python repro.py            # worker exits normally

On the hung worker, lsof -p <worker_pid> shows fd 1 and 2 are pipes, and lsof -p <sleep_pid> shows the same pipes on the child's fd 1/2. kill -9 <sleep_pid> makes the worker exit within a second.

Issue Severity

High: It blocks me from completing my task.