#1909·YOLOX

tools/train.py silently swallows training exceptions and exits with code 0 (via @logger.catch with no reraise)

Author: HikeAndMapCreated Sep 2, 2026Updated Sep 2, 2026

tools/train.py's main() is decorated with loguru's @logger.catch, which defaults to reraise=False. Any exception raised during trainer.train() - including a real training crash - is caught and logged with a full diagnostic traceback, but then swallowed: main() returns normally and the process exits with code 0. Any script, CI pipeline, or wrapper tool driving train.py and checking its exit code has no way to distinguish a genuinely successful run from one that crashed partway through.

This is made more misleading by trainer.py's own finally-block logging: after_train() always prints "Training of experiment is done and the best AP is {X}", even after a crash, using whatever best_ap had accumulated so far (often 0.00 if the crash happened before any eval ever succeeded). The log reads exactly like a normal training summary even when training never finished.

Root Cause

The problematic code block:

@logger.catch
def main(exp: Exp, args):
    ...
    trainer = exp.get_trainer(args)
    trainer.train()

logger.catch()'s default reraise=False means the decorator fully absorbs the exception after logging it - nothing propagates back up to the if __name__ == "__main__": block, so Python has no unhandled exception to report and exits 0.

Environment

  • YOLOX: main branch (tools/train.py unchanged as of this report)
  • OS: Windows 11, Python 3.12, PyTorch 2.13, loguru (current pinned version)
  • Reproduced via a real crash inside evaluate_and_save_model() (see the companion issue about the annots.pkl cache) - the exact trigger doesn't matter, any exception during trainer.train() demonstrates it

Steps to Reproduce

  1. Force an exception inside evaluate_and_save_model() (e.g. the KeyError from the stale annotation cache described in the companion issue, or any other exception during training)
  2. Run tools/train.py and let it hit that exception
  3. Check the exit code after the process ends - it is 0 despite the traceback logged to the console

Suggested Fix

Minimal, one-line change that keeps the diagnostic logging but stops hiding real failures:

@logger.catch(reraise=True)
def main(exp: Exp, args):

This still logs the full traceback via loguru, but re-raises afterward so the process exits non-zero, letting any calling script correctly detect a failed run.

Source: Megvii-BaseDetection/YOLOX