The validation mask in validation and running an inference it's not similar

I trained a nnUNetv2 model (3D fullres, fold 3) using a dataset with NoNormalization. After training, I noticed that the validation results saved in:

nnUNet_results/Dataset501_XXX/nnUNetTrainer__nnUNetPlans__3d_fullres/fold_3/validation/

are very good, as expected.

However, when I run nnUNetv2_predict using exactly the same images that were used in the validation phase (taken directly from the preprocessed dataset), the resulting segmentations are significantly different and worse.

Things I have verified: The input image is exactly the same (shape, data, header) as used during validation (from nnUNet_raw). The same trainer, same fold, same model checkpoint, and same plans.json are used. Tried with both NoNormalization and ZScoreNormalization, same issue. Inference uses: --disable_tta, --save_probabilities The shape and dtype of the prediction match the validation output. Still, comparing the validation result and the prediction result (voxel-wise), I get thousands of mismatched voxels.

Commands used:

nnUNetv2_plan_and_preprocess -d 501 --verify_dataset_integrity nnUNetv2_train 501 3d_fullres 3 -device cuda --npz

nnUNetv2_predict ^ -i "INPUT_FOLDER" ^ -o "OUTPUT_FOLDER" ^ -d 501 ^ -c 3d_fullres ^ -f 3 ^ --disable_tta ^ --save_probabilities

Hypothesis:

There might be differences in how the inference is handled internally between: The validation run inside nnUNetv2_train The inference via nnUNetv2_predict

Possible sources:

Different padding/cropping behavior Different patch stitching logic Gaussian weighting used during inference but not validation Potential randomness or floating-point instability

Question:

Is this expected behavior in nnUNetv2? Should the validation and external prediction produce exactly the same segmentation if the same image and checkpoint are used?

Any insight would be greatly appreciated.