metamotivo

Meta Motivo

Meta, FAIR

Overview

This repository provides a PyTorch implementation and pre-trained models for Meta Motivo. For details see the paper Zero-Shot Whole-Body Humanoid Control via Behavioral Foundation Models.

Features

• We provide 6 pretrained FB-CPR models for controlling the humanoid model defined in HumEnv.
• Fully reproducible scripts for evaluating the model in HumEnv.
• Fully reproducible FB-CPR training code in HumEnv for the full results in the paper, and FB training code in DMC for faster experimentation.

Installation

The project is pip installable in your environment.

pip install "metamotivo[huggingface,humenv] @ git+https://github.com/facebookresearch/metamotivo.git"

It requires Python 3.10+. Optional dependencies include humenv["bench"] and huggingface_hub for testing/training and loading models from HuggingFace.

Pretrained models

For reproducibility, we provide all the 5 models (metamotivo-S-X) we trained for producing the results in the paper, where each model is trained using a different random seed. We also provide our largest and most performant model (metamotivo-M-1), which can also be interactively tested in our demo.

Quick start

Once the library is installed, you can easily create an FB-CPR agent and download a pre-trained model from the Hugging Face hub. Note that the model is an instance of torch.nn.Module and by default it is initialized in "inference" mode (no_grad and eval mode).

We provide some simple code snippets to demonstrate how to use the model below. For more detailed examples, see our tutorials on interacting with the model, running an evaluation, and training from scratch.

Download the pre-trained models

The following code snippet shows how to instantiate the model.

from metamotivo.fb_cpr.huggingface import FBcprModel

model = FBcprModel.from_pretrained("facebook/metamotivo-S-1")

Download the buffers

For each model we provide:

• The training buffer (that can be used for inference or offline training)
• A small reward inference buffer (that contains the minimum amount of information for doing reward inference)

…

The FB-CPR model

The FB-CPR model contains several networks:

• forward net
• backward net
• critic net
• discriminator net
• actor net

We provide functions for evaluating these networks

def backward_map(self, obs: torch.Tensor) -> torch.Tensor: ...
def forward_map(self, obs: torch.Tensor, z: torch.Tensor, action: torch.Tensor) -> torch.Tensor: ...
def actor(self, obs: torch.Tensor, z: torch.Tensor, std: float) -> torch.Tensor: ...
def critic(self, obs: torch.Tensor, z: torch.Tensor, action: torch.Tensor) -> torch.Tensor: ...
def discriminator(self, obs: torch.Tensor, z: torch.Tensor) -> torch.Tensor: ...

We also provide simple functions for prompting the model and obtaining a context vector z representing the task to execute.

#reward prompt (standard and weighted regression)
def reward_inference(self, next_obs: torch.Tensor, reward: torch.Tensor, weight: torch.Tensor | None = None,) -> torch.Tensor: ...
def reward_wr_inference(self, next_obs: torch.Tensor, reward: torch.Tensor) -> torch.Tensor: ...
#goal prompt
def goal_inference(self, next_obs: torch.Tensor) -> torch.Tensor: ...
#tracking prompt
def tracking_inference(self, next_obs: torch.Tensor) -> torch.Tensor:

Once we have a context vector z we can call the actor to get actions. We provide a function for acting in the environment with a standard interface.

def act(self, obs: torch.Tensor, z: torch.Tensor, mean: bool = True) -> torch.Tensor:

Note that these functions do not allow gradient computation and use eval mode since they are expected to be used for inference (torch.no_grad() and model.eval()). For training, you should directly access the class attributes. For training we also define target networks for the forward, backward and critic networks.

Execute a policy

This is the minimal example on how to execute a random policy

…

Evaluation in HumEnv

For reproducibility of the paper, we provide a way of evaluating the models using HumEnv. We provide wrappers that can be used to interface Meta Motivo with humenv.bench reward, goal and tracking evaluation.

Here is an example of how to use the wrappers for reward evaluation:

…

You can do the same for the other evaluations provided in humenv.bench. Please refer to tutorial_benchmark.ipynb for a full evaluation loop.

Rendering a reward-based or tracking policy

We show how to render an episode with a reward-based policy.

…

It is also easy to render a policy for tracking a motion.

…

Citation

@article{tirinzoni2024metamotivo,
  title={Zero-shot Whole-Body Humanoid Control via Behavioral Foundation Models},
  author={Tirinzoni, Andrea and Touati, Ahmed and Farebrother, Jesse and Guzek, Mateusz and Kanervisto, Anssi and Xu, Yingchen and Lazaric, Alessandro and Pirotta, Matteo},
}

License

Meta Motivo is licensed under the CC BY-NC 4.0 license. See LICENSE for details.