Does libero/libero_10 out of box checkpoint give good closed loop evaluation results
Hi team,
Im trying to run a libero simulation evaluation using the publicly available checkpoint.
I followed the instructions in Isaac-GR00T/examples/LIBERO/README.md under the heading Evaluate Checkpoint and downloaded the LIBERO checkpoint using
uv run hf download nvidia/GR00T-N1.7-LIBERO --include "libero_10/config.json" "libero_10/embodiment_id.json" "libero_10/model-*.safetensors" "libero_10/model.safetensors.index.json" "libero_10/processor_config.json" "libero_10/statistics.json" --local-dir checkpoints/GR00T-N1.7-LIBERO
But after running the server and client commands i get a poor result
results: ('libero_sim/KITCHEN_SCENE3_turn_on_the_stove_and_put_the_moka_pot_on_it', [False, False, False, False, False, False, False, False, False, False], {'episode_lengths': [720, 720, 720, 720, 720, 720, 720, 720, 720, 720], 'episode_rewards': [0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0]}) success rate: 0.0
Is this expected or is the publicly available checkpoint supposed to have good evaluation results. If finetuning is required, are there any available finetuned checkpoints?
Thanks in advance
Source: NVIDIA/Isaac-GR00T