HOVER

HOVER WBC

Table of Contents

• HOVER WBC
• Table of Contents
• Overview
• Installation
• Training
  • Data Processing
    • AMASS dataset
  • Teacher Policy
  • Student Policy
  • General Remarks for Training
  • Generalist vs. Specialist Policy
• Testing
  • Play Teacher Policy
  • Play Student Policy
• Evaluation
• Overwriting Configuration Values
• Sim-to-Sim Validation
• Developing
  • Unit testing
  • Linting & Formatting
  • Set up IDE (Optional)
  • Running Scripts from an Isaac Lab Docker Container
• License
• Contributors
• Acknowledgments

Overview

This repository contains the IsaacLab extension to train neural whole-body controllers for humanoids as explained in the [OmniH2O][omnih2o_paper] and [HOVER][hover_paper] papers. For video demonstrations and to link to the original implementation in Isaac Gym, please visit the OmniH2O project website and the HOVER project website.

Installation

1. Install Isaac Lab, see the installation guide. Note: Currently HOVER has been tested with Isaac Lab versions 2.0.0. After you clone the Isaac Lab repository, check out the specific tag before installation. Also note that the rsl_rl package is renamed to rsl_rl_lib with the current v2.0.0 tag of Isaac Lab, causing installation issues. This will be fixed once a new tag is created on the Isaac Lab repo. This error would not affect this repo, as we have our own customized rsl_rl package.

   git fetch origin
   git checkout v2.0.0
   

2. Define the following environment variable to specify the path to your IsaacLab installation:

   # Set the ISAACLAB_PATH environment variable to point to your IsaacLab installation directory
   export ISAACLAB_PATH=
   

3. Clone the repo and its submodules:

   git clone --recurse-submodules 
   

4. Install this repo and its dependencies by running the following command from the root of this repo:

   ./install_deps.sh
   

Training

Data Processing

NOTE: Due to the license limitations of the AMASS dataset, we are not able to provide the retargeted dataset directly. All the following training and evaluation scripts will use the stable_punch.pkl dataset (not included as well) as a toy example. It is a small subset of the AMASS dataset where the upper body is performing punching motions. We modified the motion data to minimize the lower body's motion to create a simpler example. We suggest that users retarget a small subset of the AMASS dataset to the Unitree H1 robot and use that for trial training. The retargeting process of the whole dataset could take up to 4 days on a 32 CPU core machine. More cores will reduce the time correspondingly.

AMASS dataset

We utilize the AMASS dataset to train our models. The AMASS dataset is a comprehensive collection of motion capture (mocap) datasets. To develop control policies for a humanoid robot, it is essential to retarget the motion data in the dataset to fit the desired robot. We provide a bash script that retargets the dataset specifically for the Unitree H1 robot. This script is based on the scripts from the human2humanoid repository. Due to the limitations of the license of the AMASS dataset, we are not providing a retargeted dataset directly. To access the dataset, you will need to create an account.

To get started, follow these steps:

1. Create a folder to save the datasets in mkdir -p third_party/human2humanoid/data/AMASS/AMASS_Complete.
2. Download the dataset(s) you are interested in from the "SMPL+H G" format section on the AMASS download page and place the archive files in third_party/human2humanoid/data/AMASS/AMASS_Complete. This will take some time due to the number of datasets and the fact that apparently they don't allow parallel downloads. You don't need to extract the files manually - the script will handle that for you.
3. Download the SMPL model from this link and place the zip file third_party/human2humanoid/data/smpl.
4. Finally, run the provided script by executing ./retarget_h1.sh. The script extracts the downloaded files to desired locations, prepares necessary files and dependencies for retargeting. If you want to retarget only specific motions, you can provide a YAML file with the list of motions by running ./retarget_h1.sh --motions-file . See punch.yaml for an example. This will only process the motion files specified in the YAML file instead of the full dataset. Note that the script installs pip dependencies and might build some of them, which requires the matching version of the python-dev to be installed.
5. Before proceeding with training and evaluation, run ./install_deps.sh again to ensure the correct dependencies are installed.

The retargeted dataset will be found at third_party/human2humanoid/data/h1/amass_all.pkl. Rename it and move it to your desired location. While the exact path to the reference motion is not important, we recommend placing it in the neural_wbc/data/data/motions/ folder as the included data library will handle relative path searching, which is useful for unit testing.

For more details, refer to the human2humanoid repository.

Teacher Policy

In the project's root directory,

${ISAACLAB_PATH:?}/isaaclab.sh -p scripts/rsl_rl/train_teacher_policy.py \
    --num_envs 1024 \
    --reference_motion_path neural_wbc/data/data/motions/stable_punch.pkl

The max iteration of the teacher policy is set to 10,000,000 by default. The resulting checkpoint is stored in neural_wbc/data/data/policy/h1:teacher/ and the filename is model_.pt. The user can interrupt the training at anytime. Normally, a good policy can be trained between 50k to 80k iterations.

Student Policy

In the project's root directory,

${ISAACLAB_PATH:?}/isaaclab.sh -p scripts/rsl_rl/train_student_policy.py \
    --num_envs 1024 \
    --reference_motion_path neural_wbc/data/data/motions/stable_punch.pkl \
    --teacher_policy.resume_path neural_wbc/data/data/policy/h1:teacher \
    --teacher_policy.checkpoint model_.pt

This assumes that you have already trained the teacher policy as there is no provided teacher policy in the repo. Change the filename to match the checkpoint you trained. The exact path of the teacher policy does not matter, but it is recommended to store it in the data folder. If stored outside the data folder, you might need to provide the full path.

General Remarks for Training

• The examples above use a low number of environments as a toy demo. For good results we recommend to train with at least 4096 environments.

• The examples above use the stable_punch.pkl dataset as a toy demo. For good results we recommend to train with the full amass dataset.

• Per default the trained checkpoints are stored to logs/teacher/ or logs/student/.

• If you don't want to train from scratch you can resume training from a checkpoint using the options --teacher_policy.resume_path/--student_policy.resume_path and --teacher_policy.checkpoint/--student_policy.checkpoint. For example to resume training of the teacher use

  ${ISAACLAB_PATH:?}/isaaclab.sh -p scripts/rsl_rl/train_teacher_policy.py \
      --num_envs 10 \
      --reference_motion_path neural_wbc/data/data/motions/stable_punch.pkl \
      --teacher_policy.resume_path neural_wbc/data/data/policy/h1:teacher \
      --teacher_policy.checkpoint model_.pt
  

• Training requires a single GPU, we found the following performance when training on different GPUs:

  Teacher Training:

  GPU	Num Iterations	Time per Iteration (s)	Training Time (h)
  RTX 4090	100'000	0.84	23.3
  RTX A6000	100'000	1.90	52.8
  L40	100'000	1.61	44.6

  Student Training:

  GPU	Num Iterations	Time per Iteration (s)	Training Time (h)
  RTX 4090	10'000	0.097	0.27
  RTX A6000	10'000	0.18	0.50
  L40	10'000	0.176	0.49

Generalist vs. Specialist Policy

The codebase allows to train both generalist and specialist policies:

• Generalist policies allow to track different command configurations (or modes) with a single policy, as shown in the [HOVER][hover_paper] paper.
• Specialist policies only allow to track a specific command configuration with a single policy, as shown in the [OmniH2O][omnih2o_paper] paper.

Per default the codebase trains a specialist policy in OmniH2O mode (tracking head and hand positions).

    distill_mask_sparsity_randomization_enabled = False
    distill_mask_modes = {"omnih2o": DISTILL_MASK_MODES_ALL["omnih2o"]}

A specialist in a different mode can be trained by modifying the distill_mask_modes in the config file. For an example to train a specialist that tracks the joint angles, root linear velocity and root yaw orientation use this:

    distill_mask_sparsity_randomization_enabled = False
    distill_mask_modes = {"humanplus": DISTILL_MASK_MODES_ALL["humanplus"]}

A generalist can be trained by removing/commenting out the specialist mask modes in the config file, ie.

    distill_mask_sparsity_randomization_enabled = False
    distill_mask_modes = DISTILL_MASK_MODES_ALL

In the current implementation, we hand picked four modes that are discussed in the original paper for proof of life purposes. The user is free to add more modes to the DISTILL_MASK_MODES_ALL dictionary to make the generalist policy more general. We recommend the user to turn off sparsity randomization as the currently implemented randomization strategy (as described in the paper) might lead to motion ambiguity.

In both cases the same commands from above can be used to launch the training.

Testing

Play Teacher Policy

In the project's root directory,

${ISAACLAB_PATH:?}/isaaclab.sh -p scripts/rsl_rl/play.py \
    --num_envs 10 \
    --reference_motion_path neural_wbc/data/data/motions/stable_punch.pkl \
    --teacher_policy.resume_path neural_wbc/data/data/policy/h1:teacher \
    --teacher_policy.checkpoint model_.pt

Play Student Policy

In the project's root directory,

${ISAACLAB_PATH:?}/isaaclab.sh -p scripts/rsl_rl/play.py \
    --num_envs 10 \
    --reference_motion_path neural_wbc/data/data/motions/stable_p