[data] The val_max_samples subsample keys off data.shuffle, so data.validation_shuffle never reaches it

System Info

Reproduced by reading current main (a9f29851). Affects both trainers (trainer.use_v1 true and false).

• verl/utils/dataset/rl_dataset.py (RLHFDataset.__init__, the max_samples subsample)
• verl/trainer/ppo/utils.py (create_rl_dataset)
• verl/trainer/ppo/ray_trainer.py, verl/trainer/ppo/v1/trainer_base.py
• verl/trainer/config/data/legacy_data.yaml (shuffle, validation_shuffle)

Information

• The official example scripts
• My own modified scripts

Tasks

• An officially supported task in the examples folder (such as GLUE/SQuAD, ...)
• My own task or dataset (give details below)

Reproduction

data.val_max_samples subsamples the validation set, and whether that draw is random or a head slice is decided by self.shuffle:

# verl/utils/dataset/rl_dataset.py
self.shuffle = config.get("shuffle", False)
...
if self.max_samples > 0 and self.max_samples < total:
    if self.shuffle:
        ...
        indices = rng.choice(total, size=self.max_samples, replace=False)
    else:
        indices = np.arange(self.max_samples)

Both datasets are built from the same config.data object, so the validation dataset reads the train flag:

# verl/trainer/ppo/v1/trainer_base.py (and the V0 equivalent in ray_trainer.py)
self.train_dataset = create_rl_dataset(self.config.data.train_files, self.config.data, ...,
                                       max_samples=self.config.data.get("train_max_samples", -1))
self.val_dataset   = create_rl_dataset(self.config.data.val_files,   self.config.data, ...,
                                       max_samples=self.config.data.get("val_max_samples", -1))

data.validation_shuffle never reaches this code. It exists and is honoured — but only for the validation dataloader, at verl/trainer/ppo/ray_trainer.py:422 and verl/trainer/ppo/v1/trainer_base.py:705 (shuffle=self.config.data.get("validation_shuffle", True)).

So with the shipped defaults — shuffle: True, validation_shuffle: False (legacy_data.yaml:68 and :80) — the validation subset is drawn at random because training shuffles, while the validation loader deliberately does not shuffle. The two halves of "should validation be shuffled?" disagree, and the key the user would reach for controls only one of them.

create_rl_dataset already takes an is_train argument, which is accepted and never used:

$ grep -n "is_train" verl/trainer/ppo/utils.py

Concrete scenario

data.val_files=[aime24.parquet (30 rows), math500.parquet (500), gpqa.parquet (198)], data.val_max_samples=100, defaults otherwise. Today the 100 rows are a random draw across all three sources, so val-core/* series appear for each data_source. Setting data.validation_shuffle=false — the documented way to make validation deterministic — changes nothing about the subset.

Conversely, if the subset were made to follow validation_shuffle, the stock False would turn it into np.arange(100): since datasets.concatenate_datasets preserves file order, that is all 30 AIME rows + MATH500 rows 0-69 + zero GPQA rows, and the GPQA series silently disappears from the metrics (process_validation_metrics groups by data_source). That asymmetry is why I am filing this rather than patching it.

Expected behavior

One of these, and I do not think it is my call which:

1. The val subsample follows data.validation_shuffle. Consistent with the key's name and with the val dataloader. But on the stock validation_shuffle: False it silently changes which rows every existing val_max_samples user evaluates, per the scenario above — so it would need a release note, and arguably the YAML default should flip.
2. The val subsample is always a seeded random draw, independent of both flags, on the grounds that a validation subset should be representative regardless of whether the loader shuffles. Then shuffle/validation_shuffle govern iteration order only, and the np.arange branch stops applying to the val split.
3. Leave the behavior, fix the docs. Document that validation_shuffle affects iteration order only and that the val subsample keys off data.shuffle.

Whichever you pick, create_rl_dataset's unused is_train argument is the natural seam.

Context: this came out of #7816 / PR #7819 (the max_samples subset was drawn from OS entropy, so checkpoint resume continued on different rows). An earlier revision of that PR also rewired the val split to validation_shuffle; I removed it, because it is not needed to fix the resume bug and it changes evaluation data for every val_max_samples user. Filing it here so the decision is separable. Happy to implement whichever option you prefer.