torch.distributed.elastic.multiprocessing.errors.ChildFailedError:
Author: ghostCreated May 19, 2023Updated Oct 29, 2025
when i run this command:
torchrun --nproc-per-node 1 --master_port 25641 train.py --cfg-path train_configs/minigpt4_stage2_finetune.yaml
this error occurs, how can i fix it?
ERROR:torch.distributed.elastic.multiprocessing.api:failed (exitcode: 1) local_rank: 0 (pid: 81571) of binary: /home/tiger/miniconda3/envs/minigpt4/bin/python
Traceback (most recent call last):
File "/home/tiger/miniconda3/envs/minigpt4/bin/torchrun", line 8, in <module>
sys.exit(main())
File "/home/tiger/miniconda3/envs/minigpt4/lib/python3.9/site-packages/torch/distributed/elastic/multiprocessing/errors/__init__.py", line 346, in wrapper
return f(*args, **kwargs)
File "/home/tiger/miniconda3/envs/minigpt4/lib/python3.9/site-packages/torch/distributed/run.py", line 794, in main
run(args)
File "/home/tiger/miniconda3/envs/minigpt4/lib/python3.9/site-packages/torch/distributed/run.py", line 785, in run
elastic_launch(
File "/home/tiger/miniconda3/envs/minigpt4/lib/python3.9/site-packages/torch/distributed/launcher/api.py", line 134, in __call__
return launch_agent(self._config, self._entrypoint, list(args))
File "/home/tiger/miniconda3/envs/minigpt4/lib/python3.9/site-packages/torch/distributed/launcher/api.py", line 250, in launch_agent
raise ChildFailedError(
torch.distributed.elastic.multiprocessing.errors.ChildFailedError:
============================================================
train.py FAILED
------------------------------------------------------------
Failures:
<NO_OTHER_FAILURES>
------------------------------------------------------------
Root Cause (first observed failure):
[0]:
time : 2023-05-19_16:43:27
host : n136-117-136.byted.org
rank : 0 (local_rank: 0)
exitcode : 1 (pid: 81571)
error_file: <N/A>
traceback : To enable traceback see: https://pytorch.org/docs/stable/elastic/errors.html
========================================================================================================================Source: Vision-CAIR/MiniGPT-4