`lerobot/smolvla_libero` ships `n_action_steps=50`, which costs about 20 pp on LIBERO

Ticket Type

Bug Report (Something isn't working)

Environment & System Info

lerobot 0.6.1, hf-libero 0.1.4, robosuite 1.4.0, torch 2.11.0, Python 3.12
macOS, Apple M5 (MPS)
MuJoCo 3.3.7 for the libero_object runs, 3.8.1 for the libero_spatial runs

Description

lerobot/smolvla_libero has n_action_steps: 50 in its config, which equals chunk_size. By default lerobot-eval therefore plays the whole 50-action chunk before it looks at the scene again.

The SmolVLA paper already shows this is the worst setting on LIBERO. In Table 13, executing 50 actions averages 51.8%, against 82.8% for 10 and 80.3% for 1. HuggingFaceVLA/smolvla_libero ships n_action_steps: 1.

I checked this on the checkpoint with paired runs: same tasks and init states, only n_action_steps changed.

HuggingFaceVLA/smolvla_libero drops the same way if you force it to 50 steps: object goes from 40 to 12 of 50, spatial from 31 to 17 of 50.

Is 50 intentional? If not, setting it to 10 (or 1) in the Hub config would fix it. A SmolVLA eval command in the LIBERO docs, like the existing pi0.5 one, would also help.

Context & Reproduction

lerobot-eval --policy.path=lerobot/smolvla_libero \
  --env.type=libero --env.task=libero_object --env.task_ids="[0]" \
  --eval.n_episodes=5 --eval.batch_size=1 --seed=1000 \
  --rename_map='{"observation.images.image": "observation.images.camera1", "observation.images.image2": "observation.images.camera2"}'

Run it again with --policy.n_action_steps=10. I looped task_ids over 0–9.

• The --rename_map is this checkpoint's own camera mapping. In 0.6.1 it has to be passed by hand, because the empty CLI default replaces the saved one (fix in #4578, still open).
• The spatial runs used a small rollout loop around LeRobot's LiberoEnv on MuJoCo 3.8.1. Task 5 is broken on that version (#4390), but the gap is the same with task 5 left out (−24 pp).

Checklist

• I searched existing tickets and didn't find a duplicate.
• Latest main: tested on 0.6.1. select_action and LiberoEnv are unchanged on main as of Sep 10, and the config lives on the Hub.
• Not environment-specific: only tested on Apple MPS, not CUDA.

Additional Info / Workarounds

Workaround: pass --policy.n_action_steps=10.

A related issue, lower priority. Since #4124, --env.fps really does change the sim's control rate, and nothing warns about it. LIBERO uses delta actions, so a different rate changes how far each action moves the arm: at 10 Hz each action moved it about 1.7× as far as at 20 Hz.

With this checkpoint at 50 steps, here's how many of the 32 baseline successes survived:

Every run got about 14 s of sim time or more. robosuite rounds substeps down, so fps=30 actually runs at 31.25 Hz.

This is easy to hit by accident: lerobot/libero says fps: 10, and this checkpoint's train_config.json says env.fps: 30. A warning in LiberoEnv when fps != 20 would catch it.

Caveat: 50 episodes per cell (150 for spatial), all on Apple MPS, so the exact numbers could shift on CUDA. The direction matches the paper.

AI note: the eval runs, stats and this write-up were done with AI agents (Claude Opus 5 and Claude Sonnet 5). I reviewed it before posting.