正式实施 OpenWBT。
This project now supports teleoperation simulation in Isaac Sim and is under active maintenance. Note that official Isaac Sim is still under active development, so interfaces and features may change frequently.
The technical implementation of this project is mainly supported by R2S2. This repository implements whole body teleoperation of the Unitree G1 and H1 humanoid robot using Apple Vision Pro. The system supports both real robots and simulation environments. Only one tele-operator is needed to control the robot to perform movements such as walking, squatting, bending, grasping, and lifting, thereby significantly expanding the robot's operational capabilities and enabling it to accomplish a wider range of tasks.
In this project, the tele-operation can be split into lower-body control and upper-body control. For lower-body control, our method allows you to control the humanoid to walk or change body pose with a pair of joystick controllers. For upper-body control, the robot hand motion is derived from the human hand poses measured in VR using inverse kinematics.
See Installation.md for 1) our hardware list including the joystick and their acquisition and composition, 2) the environment configuration.
First turn on the robot:
Then use the unitree remote controller to enter the damping mode (you can also refer to Unitree official document):
For the first time you connect to the robot, you need to configure the wifi to install some dependencies. You can refer to FAQ from unitree for wifi configuration. We provide an example approach:
Modify the wifi name and corresponding password in the 01-netcfg.yaml file, and place the 01-netcfg.yaml file in /etc/netplan/ on the robot:
scp ./01-netcfg.yaml [email protected]:~/ # Transfer to the robot's root directory
# Access the remote desktop
ssh [email protected]
123 # password
1 # choose ros2 foxy(1)
sudo cp 01-netcfg.yaml /etc/netplan/
netplan apply # *You may be prompted to restart the network service during this process. Try a few more times if necessary.*
# Install dependencies
pip install pyzmq
pip install pyrealsense2
Transfer the Python file for starting the image service to the robot and run it
scp ./deploy/teleop/image_server/image_server.py [email protected]:~/image_server/
scp [email protected]:~/image_server/image_server.py ./deploy/teleop/
ssh [email protected]
123 # password
1 # choose ros2 foxy(1)
cd image_server
python image_server.py
Connect the computer to the joysticks and grant access permissions to both hands. By default, the left and right joysticks are located at /dev/ttyACM0 and /dev/ttyACM1, respectively.
sudo chmod -R 777 /dev/ttyACM0; sudo chmod -R 777 /dev/ttyACM1
Running in the Real:
source /opt/ros/foxy/setup.sh; source ~/unitree_ros2/setup.sh
python -m deploy.run_teleoperation_real --config run_teleoperation.yaml --net eno1
Running in the mujoco Simulator:
source /opt/ros/foxy/setup.sh; source ~/unitree_ros2/setup.sh
python -m deploy.run_teleoperation_mujoco --config run_teleoperation.yaml
Run in Isaac Sim (V4.5.0):
If using VR devices such as Vision Pro, run the following two commands in separate terminals to start simulation:
bash start_ik_server.sh # Terminal 1
bash start_isaacsim_vision.sh # Terminal 2
For non-VR devices, run:
bash start_isaacsim.sh
Note:
source /opt/ros/foxy/setup.sh; source ~/unitree_ros2/setup.sh
python -m deploy.run_teleoperation_real --config run_teleoperation.yaml --net eno1 --debug
Prerequisites: The teleoperation program has been executed on the user's computer, and the robot's image service has been enabled.
For the first-time connection and use in this project, please refer to the corresponding section in installation.md.
Open the browser on Apple Vision Pro and enter https://192.168.123.2:8012?ws=wss://192.168.123.2:8012.
Click "Enter VR" and allow tracking to view the first-person perspective. At this point, you can use the Vision Pro and joystick to control the actual robot.
Note:
If images fail to display in VR, please double-check the following:
power button on the right joystick briefly to make the robot enter the ready position.power button on the right joystick for 1 seconds to make the robot enter startup mode.A button on the left joystick to make the robot enter locomotion mode - use the joysticks on both the left and right joysticks to control movement and turning.D key on the left joystick to stop the robot from walking.A button on the right joystick to make the robot enter body-pose-adjustment mode - use the joysticks on both the left and right joysticks to control squatting, standing, and upper body pitch angles.D button on the right joystick to make the robot enter safe damping mode.If you find our work helpful, please cite:
@article{zhang2025unleashing,
title={Unleashing Humanoid Reaching Potential via Real-world-Ready Skill Space},
author={Zhang, Zhikai and Chen, Chao and Xue, Han and Wang, Jilong and Liang, Sikai and Liu, Yun and Zhang, Zongzhang and Wang, He and Yi, Li},
journal={arXiv preprint arXiv:2505.10918},
year={2025}
}
All code of OpenWBT is licensed under the Apache License 2.0.
This code is built upon the following open-source repositories. Please visit the respective links to view their licenses:
We evaluate our model on over 5 different Unitree G1 and at over 5 different locations. The performance is very robust. If you strictly follow our sim2real deployment code, there shouldn't be any problem. However, we still suggest adopting a conservative strategy for the first-time deployment (keep a safe distance from the robot, get ready to kill the policy at any time). If you meet any unusual robot behavior, you can directly contact us.
Though the technology behind OpenWBT is mainly supported by our recent paper R2S2, this open source project is not a strict reproduction of this paper. We add a lot of engineering tricks to make sure the policy can be deployed as robust as possible for the community and best suited for tele-operation.
If you'd like to discuss anything, feel free to send an email to [email protected] or add WeChat: zzk273939.
暂无开放 Issues，或尚未同步最近议题。