Psi0

[RSS'26] Ψ₀: An Open Foundation Model
Towards Universal Humanoid Loco-Manipulation

Contributors: [Songlin Wei](https://songlin.github.io/), [Hongyi Jing](https://hongyijing.me/), [Boqian Li](https://boqian-li.github.io/), [Zhenyu Zhao](https://zhenyuzhao.com/), [Jiageng Mao](https://pointscoder.github.io/), [Zhenhao Ni](https://nizhenhao-3.github.io/) , [Sicheng He](https://hesicheng.net/), [Jie Liu](https://jie0530.github.io/), [Xiawei Liu](https://www.xiaweiliu.com/), Kaidi Kang, Sheng Zang,[Weiduo Yuan](https://weiduoyuan.com/), [Marco Pavone](https://profiles.stanford.edu/marco-pavone), Di Huang, [Yue Wang](https://yuewang.xyz/) ------- $\Psi_0$ is an open vision-language-action (VLA) model for dexterous humanoid loco-manipulation. Our model first learns task semantics and visual representation from large-scale human egocentic videos, and then is post-trained on a smaller amount of real-world teleoperated robot data, to learn general dynamics of the embodiment. [Optional] Expand to know more about Ψ₀. Our foundation model is capable of acquiring new long-horizontal dexterous loco-manipulation skill by fine-tuning using as few as 80 trajectories. ***Our key finding is that scaling the right data in the right way.*** At the top, the $\Psi_0$ model consists of two end-to-end trained components: a vision–language backbone (System-2) and a multimodal diffusion transformer (System-1) action expert. The backbone is based on Qwen’s Qwen3-VL-2B-Instruct, which extracts vision–language features from observations and instructions. These features condition a flow-based multimodal diffusion transformer inspired by Stable **Diffusion** 3. The action expert (≈500M parameters) predicts future whole-body action chunks, enabling efficient fusion of visual, linguistic, and action representations. At the lowest level (System-0), an RL-based tracking controller executes the predicted lower-body action commands, ensuring stable and precise physical control.

## News & Updates - [x] [2026-09-13] Released new Psi-0 [checkpoints](#checkpoints) specifically trained for [SONIC](https://github.com/NVlabs/GR00T-WholeBodyControl) and its training recipes: [post-train](scripts/train/psi0/posttrain-psix-unifolm-g1-sonic1.0.sh) and [fine-tune](scripts/train/psi0/finetune-sonic-psi-dream-baseline.sh). Checkout out this [release node](examples/psi0_for_sonic.md). - [x] [2026-08-30] Released [SIMPLE SONIC wholebody training recipe](scripts/train/psi0/finetune-sonic-simple-psi0.sh). - [x] [2026-08-30] Released [docker support](#docker-support). - [x] [2026-07-14] Released DreamZero baseline for SIMPLE. - [x] [2026-06-13] Released SONIC integration for Psi-0. - [x] [2026-06-03] Psi-0 won the Best Paper Award at the 2nd 3D-LLM/VLA Workshop at CVPR 2026. ## Table of Contents - [Finetune Ψ₀ on Unitree G1 Humanoid Robot](#finetune-psi0) - [Installation](#installation) - [Data Collection](#data-collection) - [Fine-Tuning](#training-real) - [Open-Loop Evaluation](#open-loop-evaluation) - [Deployment](#deployment) - [Ψ₀ with SONIC](#psi0-sonic) - [Baselines](#baselines) - [GR00T N1.6](#groot-n16) - [OpenPi π0.5](#openpi-05) - [InternVLA-M1](#internvla-m1) - [H-RDT](#h-rdt) - [EgoVLA](#egovla) - [Diffusion Policy](#diffusion-policy) - [ACT](#act) - [Simulation ](#simulation) - [Install SIMPLE](#install-simple) - [Data Generation](#data-generation) - [Fine-Tuning](#training-sim) - [Evaluation in SIMPLE](#evaluation-in-simple) - [Reproduce Ψ₀: Pre-Training and Post-Training](#pre-post-train) - [Checkpoints](#checkpoints) - [Troubleshootings](#troubleshootings) - [Citation](#️-citation) ## Finetune Ψ₀ on Unitree G1 Humanoid Robot ### Installation Clone the project and change directory to the project root: ```bash git clone [email protected]:physical-superintelligence-lab/Psi0.git cd Psi0 ``` We use [uv](https://docs.astral.sh/uv/getting-started/installation/) to manage Python dependencies. Install `uv` if not already installed: ```bash curl -LsSf https://astral.sh/uv/install.sh | sh ``` Set up the $\Psi_0$ environment: > ℹ️ We manage the $\Psi_0$ environment and all the baselines through `uv` and they all share the same `src/` code. See [Environment Management](baselines/README.md) for more details. ``` uv venv .venv-psi --python 3.11 source .venv-psi/bin/activate GIT_LFS_SKIP_SMUDGE=1 uv sync \ --group serve \ --group viz \ --group psi \ --index-strategy unsafe-best-match \ --active uv pip install flash_attn==2.7.4.post1 --no-build-isolation ``` Test installation, a version number should be displayed. ```bash python -c "import psi;print(psi.__version__);" ``` Verify the shared `lerobot` stack is importable. ```bash python -c "from psi.data.lerobot.compat import LEROBOT_LAYOUT; print(LEROBOT_LAYOUT)" ``` #### Docker Support Pull the prebuilt image from the GitHub Container Registry: ```bash docker pull ghcr.io/physical-superintelligence-lab/psi0:latest docker tag ghcr.io/physical-superintelligence-lab/psi0:latest psi:train ``` The retag is what lets `docker-compose.yml` find it — it refers to the image as `psi:${PSI_TAG:-train}`, and `PSI_TAG` overrides only the tag, not the registry path. Use a dated tag such as `:260830` to pin an exact build. The image holds **dependencies only** — python 3.11, torch 2.7 cu12 and flash-attn in `/workspace/.venv-psi`, on a plain CUDA base. Your checkout stays outside it and is bind-mounted, so `psi` is installed editable and source edits take effect on a container restart rather than a rebuild. Serve a checkpoint with it: ```bash docker compose up serve-psi0-sonic-http ``` Every argument is overridable from the shell or `.env` (`PORT`, `RUN`, `CKPT_STEP`, `ACTION_EXEC_HORIZON`, `RTC_FLAG`, `GPUS`, ...), and `docker compose run --rm serve-psi0-sonic-http --help` lists the server's own options. If you want to build it locally: ```bash docker compose build psi # or: docker build -f scripts/train/Dockerfile -t psi:train . ``` See [scripts/train/README.md](scripts/train/README.md) for the enroot/cluster recipe and for how dependencies are pinned. [Deprecated.] Psi-0 with AMO ### Data Collection > We open-sourced all the 9 real-world tasks. You can directly download the data and jump to the [Fine-Tuning](#training-real). See the detailed teleoperation guide here: [Real-World Deployment Guide](real/README.md#real-world-deployment) #### Pre-Processing: Convert Raw Data to LeRobot Format ``` export task=Hug_box_and_move hf download USC-PSI-Lab/psi-data \ g1_real_raw/$task.zip \ --local-dir=$PSI_HOME/data/real_teleop_g1 \ --repo-type=dataset unzip $PSI_HOME/data/real_teleop_g1/g1_real_raw/$task.zip -d $PSI_HOME/data/real_teleop_g1/g1_real_raw/$task ``` You should observe similar folder structure: ``` g1_real_raw └── Hug_box_and_move ├── episode_0 │   ├── color │   │   ├── frame_000000.jpg │   │   └── ... │   └── data.json └── ... ``` Edit the task description file with the following format, eg., ``` vim scripts/data/task_description_dict.json ``` ``` { "Hug_box_and_move": "Hug box and move." } ``` Run conversion script ``` python scripts/data/raw_to_lerobot.py \ --data-root=$PWD/data/real_teleop_g1/g1_real_raw \ --work-dir=$PWD/data/real \ --repo-id=psi0-real-g1 \ --robot-type=g1 \ --task=$task ``` Calculate stats ``` python scripts/data/calc_modality_stats.py \ --work-dir=$PSI_HOME/data/real \ --task=$task ``` Create **$\Psi_0$** format stats (simply a copy for now) ``` cp $PSI_HOME/data/real/$task/meta/stats.json $PSI_HOME/data/real/$task/meta/stats_psi0.json ``` Now it's ready to finetune $\Psi_0$. > ✈️ If training env is already configured, directly launch training via `scripts/train/psi0/finetune-real-psi0.sh $task` ### Fine-Tuning > ✔️ Suppose the data is already collected and processed. Now we can proceed to fine-tune the $\Psi_0$ model. > There is a [known issue](https://github.com/physical-superintelligence-lab/Psi0/issues/3) of loading our real data, apply this fix first `python scripts/data/patch_lerobot_meta.py $PSI_HOME/data/real/$task` > Here we illustrate by using the pre-collected data from [Huggingface psi-data](https://huggingface.co/datasets/USC-PSI-Lab/psi-data/tree/main/real). Set up the environment variables following `.env.sample`. The environment variables will be loaded by the `dotenv.load_dotenv()` in python. ``` cp .env.sample .env # and edit the following env variables # HF_TOKEN= # WANDB_API_KEY= # WANDB_ENTITY= # PSI_HOME= source .env echo $PSI_HOME ``` Download the collected real-world data and extract it: ``` export task=Pick_bottle_and_turn_and_pour_into_cup hf download USC-PSI-Lab/psi-data \ real/$task.zip \ --local-dir=$PSI_HOME/data \ --repo-type=dataset unzip $PSI_HOME/data/real/$task.zip -d $PSI_HOME/data/real ``` > If you want to visualize the episode please refer to the [Data Visualization](examples/visualize.md) in the examples. Launch the training script: ``` scripts/train/psi0/finetune-real-psi0.sh $task ``` > ️ You can always change the GPUs, e.g., `CUDA_VISIBLE_DEVICES=0,1,2,3 scripts/train/...`. > ⚠️ Please try to maintain a reasonable global batch size = device batch size x number of GPUs x gradient accumulation step. We use global batch size 128 throughout all the real-world and simulation experiments. ### Open-Loop Evaluation > Follow the steps in `examples/simple/openloop_eval.ipynb` Load the training dataset, and run model inference to see how model fits the training data. ### Deployment #### Serve $\Psi_0$ (RTC mode) ```bash bash ./scripts/deploy/serve_psi0-rtc.sh ``` #### Start $\Psi_0$ Client (RTC mode) ```bash bash ./real/scripts/deploy_psi0-rtc.sh ``` For detailed real-world deployment environment setup, please also refer to the dedicated documentation: [Real-World Teleoperation Guide](real/README.md) ### Ψ₀ with SONIC [SONIC](https://github.com/NVlabs/GR00T-WholeBodyControl) is a powerful whole-body controller for humanoid robots. $\Psi_0$ now supports data collection, fine-tuning, and deployment with SONIC. Please use [our fork](https://github.com/physical-superintelligence-lab/GR00T-WholeBodyControl/tree/main) to avoid any compatibility issues. Initialize the SONIC submodule first: ```bash git submodule update --init --recursive third_party/GR00T-WholeBodyControl ``` For the full environment setup — workstation venvs, TensorRT + C++ build, PICO/XRoboToolkit, and the robot-side camera server — see the **[SONIC real-world teleoperation guide](real/SONIC/README.md)**. #### Data collection Please follow the [SONIC real-world teleoperation guide](real/SONIC/README.md#data-collection) to record demonstrations. Datasets are saved locally under `third_party/GR00T-WholeBodyControl/outputs//` in LeRobot format. #### Pre-Processing: Convert to $\Psi_0$ LeRobot Format Convert the SONIC-collected dataset into the $\Psi_0$ LeRobot format: ```bash export task= python scripts/data/raw_sonic_to_psi_lerobot.py \ --data-root=third_party/GR00T-WholeBodyControl/outputs/$task \ --work-dir=$PSI_HOME/data/sonic/lerobot \ --repo-id=$task \ --robot-type=g1 ``` Calculate stats ```bash python scripts/data/calc_modality_stats.py \ --work-dir=$PSI_HOME/data/sonic/lerobot \ --task=$task ``` Create **$\Psi_0$** format stats (simply a copy for now) ```bash cp $PSI_HOME/data/sonic/lerobot/$task/meta/stats.json $PSI_HOME/data/sonic/lerobot/$task/meta/stats_psi0.json ``` Now it's ready to fine-tune. #### Finetune $\Psi_0$ with SONIC ```bash b