[BUG] Full validation fails with a single dataset
Describe the bug
On upstream c6be9199750845bd99a7d1a438862bb9ab29b630, full validation with one dataset raises TypeError: 'int' object is not subscriptable before evaluation. This does not require Engram.
Steps/Code to reproduce bug
Set full_validation=True, multiple_validation_sets=False, and eval_iters=3 on the TP source rank.
The upstream dispatch normalizes the count to a list, but broadcasts a tensor created from the original scalar. This CPU snippet isolates the type mismatch; it does not reproduce distributed evaluation:
import torch
count = 3
eval_iters = [count]
eval_iters = torch.tensor(count).tolist() # 3, not [3]
eval_iters[0] # TypeErrorExpected behavior
Broadcast [3] and pass 3 to the evaluator. Constructing the tensor from normalized eval_iters preserves the required list shape after .tolist().
Additional context
Local tests on 8×H800 used TP=2, real NCCL broadcast and the native dispatch with a mocked evaluator. The single-dataset case failed upstream; the [2, 3] two-dataset control passed. Both passed with the one-line fix.
Environment: Python 3.12.3, PyTorch 2.12.0a0+5aff3928d8.nv26.05, CUDA 13.2. This checks dispatch, not complete dataset evaluation or official CI. The full regression tests will accompany the fix PR.
#3388 addresses a multiple-dataset deadlock; this report concerns scalar/list conversion for one dataset.
@NVIDIA/mcore-oncall
Source: NVIDIA/Megatron-LM