#4277·accelerate

`simple_launcher` discards the child's error, unlike the elastic launch path

Author: albertvillanovaCreated Sep 18, 2026Updated Sep 18, 2026

System Info

accelerate 1.12.0, and unchanged on main at b795b4838eb33bde60eb398649c4ab006874e3e9

Information

  • The official example scripts
  • My own modified scripts

Tasks

  • One of the scripts in the examples/ folder of Accelerate or an officially supported no_trainer script in the examples folder of the transformers repo (such as run_no_trainer_glue.py)
  • My own task or dataset (give details below)

Reproduction

simple_launcher runs the training script with subprocess.Popen and, on a non-zero exit, raises a CalledProcessError built from the return code and the command alone: https://github.com/huggingface/accelerate/blob/b795b4838eb33bde60eb398649c4ab006874e3e9/src/accelerate/commands/launch.py#L992-L1001

The child's stderr is inherited rather than read, and neither output nor stderr is passed to the exception, so a caller that catches it learns only that something exited non-zero.

python
import subprocess
import sys

from accelerate.commands.launch import launch_command, launch_command_parser

args = launch_command_parser().parse_args(["--cpu", "--num_processes", "1", sys.argv[1]])
try:
    launch_command(args)
except subprocess.CalledProcessError as e:
    print(f"parent sees : {type(e).__name__}: {e}")
    print(f"e.stderr    : {e.stderr!r}")
    print(f"e.output    : {e.output!r}")

Run against a boom.py whose only line is raise ValueError("the real cause"):

Traceback (most recent call last):
  File "boom.py", line 1, in <module>
    raise ValueError("the real cause")
ValueError: the real cause

parent sees : CalledProcessError: Command '['.../bin/python', 'boom.py']' returned non-zero exit status 1.
e.stderr    : None
e.output    : None

Expected behavior

The multi-process path already solves this. multi_gpu_launcher delegates to torch.distributed.run, and torch elastic propagates the child's root exception to the parent through error files, so the parent raises a ChildFailedError carrying the child's traceback in a "Root Cause" section. Its docstring gives the rationale: the parent is "a simple nanny process", so "child errors should be propagated to the scheduler for accurate root cause diagnostics". simple_launcher is the same nanny, and it would be good if it gave the same guarantee.

This matters for callers that handle the exception programmatically rather than reading a terminal. Concretely, the TRL CLI tests invoke trl sft, which reaches simple_launcher, and the suite retries known transient infrastructure failures by matching the exception type and message. When a child dies of GPU memory pressure from parallel test workers, all that reaches the matcher is CalledProcessError: Command '[...]' returned non-zero exit status 1., indistinguishable from a genuine regression, so the retry never fires and the run fails permanently on a transient condition. The child's traceback does reach the terminal, but a thousand lines above the failure report and interleaved with the output of the other workers, which is of no use to a programmatic caller.

Two ways to close the gap, in increasing order of effort:

  1. Drain the child's stderr through a pipe, write it through to the parent's stderr as it arrives, and pass the tail as stderr= on the CalledProcessError. Live output is preserved and the cause survives.
  2. Wrap the launched script with the elastic error-file mechanism, so simple_launcher reports failures the way multi_gpu_launcher already does.

I am happy to open a PR for either one if you have a preference.