--val --val_best fails when checkpoint_final is missing
Author: hajteyibCreated Jul 20, 2026Updated Aug 4, 2026
Labelsbugneeds-maintainer
Hi,
We had an interrupted training where checkpoint_best.pth was available but there was no checkpoint_final.pth.
I wanted to run the validation with:
nnUNetv2_train DATASET 3d_fullres FOLD --val --val_bestI found #2459 where this command was suggested for this situation. But on the current master (27f878d) I get:
RuntimeError: Cannot run validation because the training is not finished yet I also made a small test without data or GPU:
from pathlib import Path
from tempfile import TemporaryDirectory
from nnunetv2.run.run_training import maybe_load_checkpoint
class Trainer:
def __init__(self, folder):
self.output_folder = folder
def load_checkpoint(self, path):
print(path)
with TemporaryDirectory() as folder:
Path(folder, "checkpoint_best.pth").touch()
maybe_load_checkpoint(Trainer(folder), False, True)It looks like maybe_load_checkpoint checks for checkpoint_final.pth first and raises the error. The code for val_with_best is called only after this, so it never reaches checkpoint_best.pth.
I know that stopping the training early is not the recommended nnU-Net use. I am just not sure if this behavior is expected.
Would it make sense that:
--valrequirescheckpoint_final.pth--val --val_bestrequirescheckpoint_best.pth
If yes, I can propose a small PR with tests.
Thank you!
Source: MIC-DKFZ/nnUNet