[BUG] Full validation fails with a single dataset

Author: jeyaletheiaCreated Sep 9, 2026Updated Sep 18, 2026
Labelscommunity-request

Describe the bug

On upstream c6be9199750845bd99a7d1a438862bb9ab29b630, full validation with one dataset raises TypeError: 'int' object is not subscriptable before evaluation. This does not require Engram.

Steps/Code to reproduce bug

Set full_validation=True, multiple_validation_sets=False, and eval_iters=3 on the TP source rank.

The upstream dispatch normalizes the count to a list, but broadcasts a tensor created from the original scalar. This CPU snippet isolates the type mismatch; it does not reproduce distributed evaluation:

python
import torch

count = 3
eval_iters = [count]
eval_iters = torch.tensor(count).tolist()  # 3, not [3]
eval_iters[0]  # TypeError

Expected behavior

Broadcast [3] and pass 3 to the evaluator. Constructing the tensor from normalized eval_iters preserves the required list shape after .tolist().

Additional context

Local tests on 8×H800 used TP=2, real NCCL broadcast and the native dispatch with a mocked evaluator. The single-dataset case failed upstream; the [2, 3] two-dataset control passed. Both passed with the one-line fix.

Environment: Python 3.12.3, PyTorch 2.12.0a0+5aff3928d8.nv26.05, CUDA 13.2. This checks dispatch, not complete dataset evaluation or official CI. The full regression tests will accompany the fix PR.

#3388 addresses a multiple-dataset deadlock; this report concerns scalar/list conversion for one dataset.

@NVIDIA/mcore-oncall