RDT2

Table of Contents

• Table of Contents
• Overview
• Updates
• Requirements
• Installation
• Model Checkpoints
• Running Inference for a Pre-Trained Model
  • 1. [IMPORTANT] Hard-ware Set up and Calibration
  • 2. Run Inference
• Fine-Tuning Models on Your Own Data
  • 1. Convert your data to WebDataset shards
  • 2. Defining training configs and running training
  • 3. Run training
    • RDT2-VQ
    • RDT2-FM
  • Precision Settings
• Troubleshooting

Overview

RDT2, the sequel to RDT-1B, is the first foundation model that can achieve zero-shot deployment on unseen embodiments for simple open-vocabulary tasks like picking, placing, shaking, wiping, etc. This milestone was made possible by multifaceted efforts:

• We redesigned the UMI hardware by applying higher-strength materials and more precise tracking methods, ensuring its reliability for large-scale data collection.
• We collected 10,000+ hours of human manipulation videos in 100+ different indoor scenes, covering the majority of household tasks that a gripper can do.

Currently, this repo contains models:

• the RDT2-VQ, an auto vision-language-action model (VLA) which employs Residual VQ as the action tokenizer, is adapted from Qwen2.5-VL-7B-Instruct with our UMI dataset, enabling superior zero-shot instruction-following capability.
• the RDT2-FM, an improved RDT model as action expert with flow-matching objective, running with much lower inference latency.

For all models, we provide checkpoints and examples for using them out of the box or fine-tuning them to your own datasets. Currently, we have verified the efficay of our models on platforms including Bimanual UR5e and Bimanual Franka Research 3, and we are optimistic will able to deploy them successfully on more platforms in the future by following our guidelines.

Updates

• [Sept 2025] We released RDT2-VQ & RDT2-FM, the sequel of RDT-1B with better open-world generalization and zero-shot deployment on unseen embodiments.
• [Feb 2026] We released the arXiv paper.

Requirements

To run the models in this repository, you will need an NVIDIA GPU with at least the following specifications. These estimations assume a single GPU, but you can also use multiple GPUs with model parallelism or offload into CPU to reduce per-GPU memory. Since RDT2 is based on Qwen2.5-VL-7B, you basiclly need to follow the hard-ware requirements for Qwen2.5-VL-7B:

As for zero-shot deployment, you need to purchase the designated end effector and camera, and 3D print the corresponding camera stand and flange according to Harware Set up and Calibration.

The repo has been tested with Ubuntu 24.04, we do not currently support other operating systems.

Installation

Clone this repo and create a conda environment:

…

Model Checkpoints

We provide multiple VLA model checkpoints with capabilities to deploy on various robot platforms and simple vocabulary tasks. If you want to deploy on your own robot platform with other end effectors and cameras, you can fine-tune from the base model.

Running Inference for a Pre-Trained Model

1. [IMPORTANT] Hard-ware Set up and Calibration

1. Acquire deployment hardware according to our Hardware Guide.

2. Set up Robots

• 2.1 Set up UR5e

  • Obtain IP address and update configs/robots/eval_bimanual_ur5e_config.yaml/robots/robot_ip.
  • In Installation > Payload
    • Set mass to 0.82 kg
    • Set Inertia Matrix to
      python复制

      [0.001106, 0, 0,
       0, 0.001106, 0,
       0, 0, 0.001106]

    • Set speed to 30%(recommened)

• 2.2 Set up Franka FR3

  • Obtain IP address and update configs/robots/eval_bimanual_fr3_config.yaml/robots/robot_ip.
  • On the Franka interface website
    • Set gripper mass to 1.9 kg
    • Set Inertia Tensor to
      python复制

      [0.001, 0, 0,
       0, 0.001, 0,
       0, 0, 0.001]

1. Set up camera

   • Download SDK from HikRobot website and install all the .deb files.
   • Run cd /opt/MVS/bin && ./MVS.sh. Select your camera, and set Acquisition Control -> Exposure Time to 20000.

2. Calibrate your robot to tracker's tcp space

• Follow Setup Instructions For Calibration in Hardware Guide.
• Set up Vive Tracker according to this tutorial ->Software Setup Tutorial ->VIVE tracker setup
• Run the following code to calibrate robot tcp space to tracker's space.
• IMPORTANT: This script makes the robot perform small-amplitude sinusoidal motions; before running the script, please ensure the robot is in a safe position and the workspace is free of obstacles.
  bash复制

  python deploy/calibration/calibrate_franka.py --franka_ip  --franka_port  # if using Franka Research 3
  # or
  python deploy/calibration/calibrate_ur5e.py --ur5e_ip  # if using UR5e

• After calibration, run the following script to obtain the calibration matrix:
  bash复制

  python deploy/calibration/compute_calibration_matrix.py

  Then paste the calibration matrix to eval_bimanual_ur5e_config.yaml/tx_tracker_to_tcp (or eval_bimanual_fr3_config.yaml/tx_tracker_to_tcp if using FR3).

2. Run Inference

Our pre-trained model checkpoints can be run with a few lines of code (here our RDT2-VQ model):

…

And you can also test the RDT2-FM with the following code:

…

We provide detailed step-by-step examples for running inference of our pre-trained checkpoints on Bimanual UR5e and Bimanual Franka Research 3 robots.

IMPORTANT: If the inference success rate is still low after checking all the settings, configurations and calibrations, you can refer to the deployment tips for help.

Fine-Tuning Models on Your Own Data

We will fine-tune the RDT2 models on the example dataset from Bimanual UR5e as a running example for how to fine-tune a base model on your own data. We will explain three steps:

1. Convert your data to a webdataset shards (which we use for training for high-efficent IO)
2. Define training configs
3. Run training

1. Convert your data to WebDataset shards

You should convert to a processed webdataset shards, with the following structure:

…

Moreover, we provde processed example data collected with Bimanual UR5e on huggingface. You can download it and use it directly.

2. Defining training configs and running training

Define your dataset config following format in configs/datasets/example.yaml

# Define your dataset name here
name:  # e.g. bimanual/ur_example
type: single
shards_dir:  # e.g. /ssd/rdt2/bimanual_fold_cloth/shards 
kwargs:
  instruction_path:  # e.g. /ssd/rdt2/ur_example/instruction.json
  normalizer_path:  # e.g. /ssd/rdt2/umi_normalizer_wo_downsample_indentity_rot.pt

For the provided example data, its corresponding config is in configs/datasets/example.yaml. Remember to replace the and with your own path for downloading.

3. Run training

RDT2-VQ

Currently, we support the following fine-tuning methods:

• DeepSpeed training
• LoRA (low-rank adaptation) training

Since RDT2-VQ is based on Qwen2.5-VL, you are free to apply using other techniques including (e.g., fsdp, quantization) by following Qwen2.5-VL's fine-tunig practices. We provide example fine-tuning scripts for full-parameter and LoRA fine-tuning, which you can directly use to kick off your own training.

To provide a better understanding, we elaborate the line-by-line explanation of the full-parameter fine-tuning script (scripts/finetune_full_param.sh) with our example data:

…

Although our RVQ demonstrates high generalization among both hand-held gripper data and real robot data. If you want to fine-tune on your own data with our Residual VQ as action tokenzer, we sincerely suggest you firstly to check the statistics of your data are within the bound of our Residual VQ, and then test the reconstruction error of your data.

RDT2-FM

Currently, we support fine-tuning RDT2-FM's Action Expert with DeepSpeed: We provide example fine-