Cannot reproduce the GR00T 1.6 vs 1.7 RoboCasa (Panda Omron) README table from the documented eval command; the two columns appear to come from different protocols
The doc issue
examples/robocasa/README.md reports a 24-task GR00T 1.6 vs 1.7 table (average 66.22% → 70.8%; 9 tasks lower, the largest OpenDoubleDoor 39.0 → 25.0). Three things I can't reconcile from the README itself:
1. The documented eval command is --n-episodes 10 --n-envs 5, but neither column is consistent with n=10. Every 1.7 value is a multiple of 5% (except PnPMicrowaveToCounter at 19.0%), which fits n=20. The 1.6 column sits on a 0.5% grid (63.5, 98.5, 88.5, …), which needs n=200 — and two of its values, 81.1% and 63.2%, fit neither 200 nor any common count. Were the two columns evaluated at different episode counts, or is one of them an average over several runs? n_envs isn't stated either, and #600 shows results move between n_envs=1 and n_envs=50 because invalid episodes are dropped silently.
2. The 1.6 row links to the base nvidia/GR00T-N1.6-3B, the 1.7 row says "finetuned checkpoint", and the README states "RoboCasa Panda Omron is not present in the base model checkpoint; use a checkpoint finetuned with ROBOCASA_PANDA_OMRON." Is the 1.6 column the base model, or a 1.6 checkpoint finetuned with the same recipe (MAX_STEPS=60000, GLOBAL_BATCH_SIZE=512)? If it's the base model, the table compares pretraining coverage against finetuning rather than 1.6 against 1.7. This was asked in #506 (January, no reply) and again in the April follow-up on #552, which is still open.
3. At n=20, a 14-point drop is about 1.4 standard errors. The binomial SE on a 25% rate at 20 episodes is ±9.7 pp. If the 1.7 column is n=20, none of the nine per-task drops separates from sampling noise — and at that count neither does the +4.6 average.
For SimplerEnv, #685 turned out to be a different seed distribution behind the README numbers, and the table was corrected in 65cc4a1. This looks like the same shape of problem.
Suggest a potential alternative/fix
Add to the table, or a line under it: episodes per task and n_envs for each column; the seed / init-state handling; and whether the 1.6 column is the base checkpoint or finetuned, with the recipe if so. Happy to open the PR once the numbers are confirmed.
Source: NVIDIA/Isaac-GR00T