`simple_launcher` discards the child's error, unlike the elastic launch path
System Info
accelerate 1.12.0, and unchanged on main at b795b4838eb33bde60eb398649c4ab006874e3e9Information
- The official example scripts
- My own modified scripts
Tasks
- One of the scripts in the examples/ folder of Accelerate or an officially supported
no_trainerscript in theexamplesfolder of thetransformersrepo (such asrun_no_trainer_glue.py) - My own task or dataset (give details below)
Reproduction
simple_launcher runs the training script with subprocess.Popen and, on a non-zero exit, raises a CalledProcessError built from the return code and the command alone: https://github.com/huggingface/accelerate/blob/b795b4838eb33bde60eb398649c4ab006874e3e9/src/accelerate/commands/launch.py#L992-L1001
The child's stderr is inherited rather than read, and neither output nor stderr is passed to the exception, so a caller that catches it learns only that something exited non-zero.
import subprocess
import sys
from accelerate.commands.launch import launch_command, launch_command_parser
args = launch_command_parser().parse_args(["--cpu", "--num_processes", "1", sys.argv[1]])
try:
launch_command(args)
except subprocess.CalledProcessError as e:
print(f"parent sees : {type(e).__name__}: {e}")
print(f"e.stderr : {e.stderr!r}")
print(f"e.output : {e.output!r}")Run against a boom.py whose only line is raise ValueError("the real cause"):
Traceback (most recent call last):
File "boom.py", line 1, in <module>
raise ValueError("the real cause")
ValueError: the real cause
parent sees : CalledProcessError: Command '['.../bin/python', 'boom.py']' returned non-zero exit status 1.
e.stderr : None
e.output : NoneExpected behavior
The multi-process path already solves this. multi_gpu_launcher delegates to torch.distributed.run, and torch elastic propagates the child's root exception to the parent through error files, so the parent raises a ChildFailedError carrying the child's traceback in a "Root Cause" section. Its docstring gives the rationale: the parent is "a simple nanny process", so "child errors should be propagated to the scheduler for accurate root cause diagnostics". simple_launcher is the same nanny, and it would be good if it gave the same guarantee.
This matters for callers that handle the exception programmatically rather than reading a terminal. Concretely, the TRL CLI tests invoke trl sft, which reaches simple_launcher, and the suite retries known transient infrastructure failures by matching the exception type and message. When a child dies of GPU memory pressure from parallel test workers, all that reaches the matcher is CalledProcessError: Command '[...]' returned non-zero exit status 1., indistinguishable from a genuine regression, so the retry never fires and the run fails permanently on a transient condition. The child's traceback does reach the terminal, but a thousand lines above the failure report and interleaved with the output of the other workers, which is of no use to a programmatic caller.
Two ways to close the gap, in increasing order of effort:
- Drain the child's stderr through a pipe, write it through to the parent's stderr as it arrives, and pass the tail as
stderr=on theCalledProcessError. Live output is preserved and the cause survives. - Wrap the launched script with the elastic error-file mechanism, so
simple_launcherreports failures the waymulti_gpu_launcheralready does.
I am happy to open a PR for either one if you have a preference.
Source: huggingface/accelerate