Negative delta_indices (history frames) silently read from the end of the episode during training

Describe the bug

extract_step_data (gr00t/data/dataset/sharded_single_step_dataset.py) computes indices_to_load = [step_index + delta for delta in config.delta_indices] and, when allow_padding=False, hands the list straight to DataFrame.iloc. pandas accepts negative positions and counts them from the end of the episode, so any history offset (negative delta_index) at the start of an episode silently returns frames from the episode's last rows.

DataConfig.allow_padding defaults to False and launch_finetune.py never sets it (there is no CLI flag for it), so this is the default training path. The in-repo oxe_droid_relative_eef_relative_joint config uses video: delta_indices=[-15, 0] (gr00t/configs/data/embodiment_configs.py:30), and examples/DROID/README.md documents finetuning with --embodiment-tag OXE_DROID_RELATIVE_EEF_RELATIVE_JOINT. For every DROID episode, training steps 0–14 are therefore paired with a "previous frame" taken from the final 15 frames of the episode. ShardedSingleStepDataset.get_effective_episode_length only subtracts the action horizon, so the sharder schedules all of those steps.

This is a regression. N1.5's gr00t/data/dataset.py padded out-of-range history via retrieve_data_and_pad(..., padding_strategy="first_last"), which mapped front indices to array[0]. The N1.6 release commit 4e62473 (#456) removed that file and introduced the unguarded iloc path.

Reproduction

Uses only demo_data/droid_sample from this repo; CPU only, no model weights and no GPU needed. The index list is computed once per modality and applied identically to every modality, so the shipped video offsets [-15, 0] are mirrored onto the state stream to keep the snippet runnable without torchcodec.

import numpy as np
from gr00t.configs.data.embodiment_configs import MODALITY_CONFIGS
from gr00t.data.dataset.lerobot_episode_loader import LeRobotEpisodeLoader
from gr00t.data.dataset.sharded_single_step_dataset import (
    ShardedSingleStepDataset,
    extract_step_data,
)
from gr00t.data.types import EmbodimentTag, ModalityConfig

tag = EmbodimentTag.OXE_DROID_RELATIVE_EEF_RELATIVE_JOINT
cfg = dict(MODALITY_CONFIGS[tag.value])
cfg.pop("video")
cfg["state"] = ModalityConfig(delta_indices=[-15, 0], modality_keys=cfg["state"].modality_keys)

ep = LeRobotEpisodeLoader("demo_data/droid_sample", cfg)[1]  # 266 frames
print("episode length:", len(ep))
for step in (0, 5, 14, 15):
    d = extract_step_data(ep, step, cfg, tag, allow_padding=False)
    hist = d.states["joint_position"][0]  # the "-15" frame
    print(
        step,
        "is_first_frame:", np.allclose(hist, ep["state.joint_position"].iloc[0]),
        "is_episode_tail_row:", np.allclose(hist, ep["state.joint_position"].iloc[step - 15]),
    )

ds = ShardedSingleStepDataset(
    "demo_data/droid_sample", tag, cfg, shard_size=64, episode_sampling_rate=1.0
)
print(
    "scheduled steps < 15:",
    sorted(int(s) for sh in ds.sharded_episodes for e, steps in sh if e == 1 for s in steps if s < 15),
)

Observed output on main (51d4c89):

episode length: 266
0 is_first_frame: False is_episode_tail_row: True     # history frame == row 251 of 266
5 is_first_frame: False is_episode_tail_row: True     # row 256
14 is_first_frame: False is_episode_tail_row: True    # row 265
15 is_first_frame: True  is_episode_tail_row: True    # row 0, first in-range step
scheduled steps < 15: [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14]

Expected: the history frame for steps 0–14 should be frame 0 of the episode (or those steps should be excluded), not rows 251–265.

Episodes 0 and 2 of the same sample behave identically (rows 152/166 of 167 and 396/410 of 411 for steps 0/14).

Scope

• Silent — no error, no warning; the loss looks normal.
• Bounded — only the first |min(delta_indices)| scheduled steps of each episode (15 of 227 effective steps on the 266-frame sample, ~6% on DROID-length episodes), and only the history frame of those samples. The current frame, state, actions and language are correct.
• Train/inference mismatch — both in-tree deployment paths pad history by repeating the first observation: MultiStepWrapper.reset builds deque([obs] * (max_steps_needed + 1)) (gr00t/eval/sim/wrapper/multistep_wrapper.py:285), and examples/DROID/main_gr00t.py reads frame_buffer[0] while the buffer fills. The model is trained on episode-tail frames for those steps and sees repeated first frames at deployment.
• Offline eval hits the same wrong frame at step 0: gr00t/eval/open_loop_eval.py, scripts/deployment/standalone_inference_script.py and scripts/deployment/export_onnx_n1d7.py rely on the default; scripts/deployment/benchmark_inference.py, getting_started/GR00T_inference.ipynb and scripts/deployment/GR00T_inference_timing.ipynb pass allow_padding=False explicitly.
• Live deployment (policy server, MultiStepWrapper, the DROID client) never calls extract_step_data and is not affected.
• Not only DROID: the released nvidia/GR00T-N1.7-3B processor_config.json carries eight pretrain tags with history windows ([-15, 0], [-20, 0], [-30, 0]), and #152 / #368 point users at [-1, 0] for history, so custom embodiments with history frames take the same path. The released nvidia/GR00T-N1.7-DROID finetune uses video: [0], so that checkpoint did not exercise it.

I have not been able to measure whether this measurably degrades a finetuned policy — that needs a GPU and a full DROID run.

Possibly related, but not duplicates

• #743 (open) re-implements the same arithmetic in LeRobotEpisodeLoaderFaster; its test_no_padding_keeps_negative_indices asserts the negative pass-through, so it would need the same clamp.
• #152 / #368 — questions about history horizons; the wraparound is not mentioned.
• getting_started/data_config.md currently states that "no current N1.7 embodiment config uses" negative indices, which embodiment_configs.py:30 contradicts.

Suggested fix

When allow_padding=False, clamp negative observation indices to frame 0 (matching the inference-side padding and the old N1.5 behaviour) and raise on any other out-of-range index instead of letting iloc wrap; leave allow_padding=True unchanged. A PR with tests follows.

A one-line alternative — defaulting DataConfig.allow_padding to True — produces identical results on every sharder-scheduled step, if you would rather not change the False semantics. Changing the shipped oxe_droid_relative_eef_relative_joint config to video: [0] (matching the released DROID checkpoint) would sidestep it for that tag only, leaving custom history configs affected.

Versions

Reproduced on main @ 51d4c89. The repro needs only pandas + pyarrow + numpy (no GPU, no model weights, no torchcodec), so this is a minimal environment rather than a full uv sync --all-extras install:

Python 3.12.10
numpy==2.5.2
pandas==3.0.5
pyarrow==25.0.1
scipy==1.18.1
torch==2.14.0+cpu
transformers==5.16.1
pytest==9.1.1
ruff==0.16.5
tyro==1.0.16

The negative-iloc semantics that cause this are stable across pandas releases, so the pandas version should not matter.