tools/train.py silently swallows training exceptions and exits with code 0 (via @logger.catch with no reraise)
tools/train.py's main() is decorated with loguru's @logger.catch, which defaults to reraise=False. Any exception raised during trainer.train() - including a real training crash - is caught and logged with a full diagnostic traceback, but then swallowed: main() returns normally and the process exits with code 0. Any script, CI pipeline, or wrapper tool driving train.py and checking its exit code has no way to distinguish a genuinely successful run from one that crashed partway through.
This is made more misleading by trainer.py's own finally-block logging: after_train() always prints "Training of experiment is done and the best AP is {X}", even after a crash, using whatever best_ap had accumulated so far (often 0.00 if the crash happened before any eval ever succeeded). The log reads exactly like a normal training summary even when training never finished.
Root Cause
The problematic code block:
@logger.catch
def main(exp: Exp, args):
...
trainer = exp.get_trainer(args)
trainer.train()logger.catch()'s default reraise=False means the decorator fully absorbs the exception after logging it - nothing propagates back up to the if __name__ == "__main__": block, so Python has no unhandled exception to report and exits 0.
Environment
- YOLOX: main branch (tools/train.py unchanged as of this report)
- OS: Windows 11, Python 3.12, PyTorch 2.13, loguru (current pinned version)
- Reproduced via a real crash inside
evaluate_and_save_model()(see the companion issue about theannots.pklcache) - the exact trigger doesn't matter, any exception duringtrainer.train()demonstrates it
Steps to Reproduce
- Force an exception inside
evaluate_and_save_model()(e.g. theKeyErrorfrom the stale annotation cache described in the companion issue, or any other exception during training) - Run
tools/train.pyand let it hit that exception - Check the exit code after the process ends - it is
0despite the traceback logged to the console
Suggested Fix
Minimal, one-line change that keeps the diagnostic logging but stops hiding real failures:
@logger.catch(reraise=True)
def main(exp: Exp, args):This still logs the full traceback via loguru, but re-raises afterward so the process exits non-zero, letting any calling script correctly detect a failed run.
Source: Megvii-BaseDetection/YOLOX