3D-Diffusion-Policy

3D Diffusion Policy

Project Page | arXiv | Twitter | Data

Yanjie Ze*, Gu Zhang*, Kangning Zhang, Chenyuan Hu, Muhan Wang, Huazhe Xu

Robotics: Science and Systems (RSS) 2024

3D Diffusion Policy (DP3) is a universal visual imitation learning algorithm that marries 3D visual representations with diffusion policies, achieving surprising effectiveness in diverse simulated and real-world tasks, including both high-dimensional and low-dimensional control tasks, with a practical inference speed.

Small tips for using DP3:

• Try longer prediction horizon and longer action horizon, e.g., 8/16/32, to get better results.
• Try to use global position instead of relative position as action space.

Applications and extensions of DP3 from the community:

• arXiv 2025.10, RL-100: Performant Robotic Manipulation with Real-World Reinforcement Learning, where DP3 shows 100% manipulation success rate after offline RL and online RL finetuning.
• arXiv 2025.09, ManiFlow: A General Robot Manipulation Policy via Consistency Flow Training, where DP3 is improved via new backbones and flow matching objective.
• arXiv 2025.07, HumanoidGen: Data Generation for Bimanual Dexterous Manipulation via LLM Reasoning, where DP3 shows superior performance over DP on bimanual humanoid manipulation tasks in simulation.
• arXiv 2025.06, Vision in Action: Learning Active Perception from Human Demonstrations, where DP3 serves as a strong baseline in active perception tasks.
• arXiv 2025.06, ClutterDexGrasp: A Sim-to-Real System for General Dexterous Grasping in Cluttered Scenes, where DP3 shows impressive dex grasping capbilities by distilling an RL teacher policy.
• arXiv 2025.05, H3DP: Triply-Hierarchical Diffusion Policy for Visuomotor Learning, where DP3 shows significantly better results over DP and DP w/ depth.
• arXiv 2025.03, PointVLA: Injecting the 3D World into Vision-Language-Action Models, where iDP3 Encoder is used in training a powerful 3D VLA.
• arXiv 2025.03, FP3: A 3D Foundation Policy for Robotic Manipulation, where DP3 shows great generalizability abilities comparable to a large-scale trained 3D policy FP3.
• arXiv 2025.02, DemoGen: Synthetic Demonstration Generation for Data-Efficient Visuomotor Policy Learning, where DP3 can achieve great real-world abilities with generated demos.
• arXiv 2024.11, Motion Before Action: Diffusing Object Motion as Manipulation Condition, where DP3 shows improved effectiveness with a Motion-Before-Action module.
• arXiv 2024.10, Generalizable Humanoid Manipulation with 3D Diffusion Policies, where improved DP3 shows effectiveness in humanoid manipulation tasks and impressive generalization abilities across scenes.
• arXiv 2024.09, Mamba Policy: Towards Efficient 3D Diffusion Policy with Hybrid Selective State Models, where the backbone of DP3 is replaced with the Mamba archiecture.
• arXiv 2024.09, RoboTwin: Dual-Arm Robot Benchmark with Generative Digital Twins, where DP3 is well benchmarked on 6 new simulated bimanual tasks.
• arXiv 2024.08, ACE: A Cross-Platform Visual-Exoskeletons System for Low-Cost Dexterous Teleoperation, where DP3 shows effectiveness in bimanual dexterous tasks.
• arXiv 2024.07, Bunny-VisionPro: Real-Time Bimanual Dexterous Teleoperation for Imitation Learning, where DP3 shows effectiveness in bimanual long-horizon tasks.
• arXiv 2024.07, EquiBot: SIM(3)-Equivariant Diffusion Policy for Generalizable and Data Efficient Learning, where DP3 is able to fold clothes with high success rates.
• arXiv 2024.06, ManiCM: Real-time 3D Diffusion Policy via Consistency Model for Robotic Manipulation, where DP3 is accelerated via consistency model.
• arXiv 2024.03, Learning Visual Quadrupedal Loco-Manipulation from Demonstrations, where DP3 is used as the high-level planner.

Benchmark of DP3

Simulation environments. We provide dexterous manipulation environments and expert policies for Adroit, DexArt, and MetaWorld in this codebase (3+4+50=57 tasks in total). the 3D modality generation (depths and point clouds) has been incorporated for these environments.

Real-world robot data is also provided here.

Algorithms. We provide the implementation of the following algorithms:

• DP3: dp3.yaml
• Simple DP3: simple_dp3.yaml

Among these, dp3.yaml is the proposed algorithm in our paper, showing a significant improvement over the baselines. During training, DP3 takes ~10G gpu memory and ~3 hours on an Nvidia A40 gpu, thus it is feasible for most researchers.

simple_dp3.yaml is a simplified version of DP3, which is much faster in training (1~2 hour) and inference (25 FPS) , without much performance loss, thus it is more recommended for robotics researchers.

Installation

See INSTALL.md for installation instructions.

See ERROR_CATCH.md for error catching I personally encountered during installation.

Data

You could generate demonstrations by yourself using our provided expert policies. Generated demonstrations are under $YOUR_REPO_PATH/3D-Diffusion-Policy/data/.

• Download Adroit RL experts from OneDrive or GoogleDrive, unzip it, and put the ckpts folder under $YOUR_REPO_PATH/third_party/VRL3/.
• Download DexArt assets from Google Drive and put the assets folder under $YOUR_REPO_PATH/third_party/dexart-release/.

Note: since you are generating demonstrations by yourselves, the results could be slightly different from the results reported in the paper. This is normal since the results of imitation learning highly depend on the demonstration quality. Please re-generate demonstrations if you encounter some bad demonstrations and no need to open a new issue.

️ Usage

Scripts for generating demonstrations, training, and evaluation are all provided in the scripts/ folder.

The results are logged by wandb, so you need to wandb login first to see the results and videos.

For more detailed arguments, please refer to the scripts and the code. We here provide a simple instruction for using the codebase.

1. Generate demonstrations by gen_demonstration_adroit.sh and gen_demonstration_dexart.sh. See the scripts for details. For example:

   bash复制

   bash scripts/gen_demonstration_adroit.sh hammer

   This will generate demonstrations for the hammer task in Adroit environment. The data will be saved in 3D-Diffusion-Policy/data/ folder automatically.

2. Train and evaluate a policy with behavior cloning. For example:

   bash复制

   bash scripts/train_policy.sh dp3 adroit_hammer 0112 0 0

   This will train a DP3 policy on the hammer task in Adroit environment using point cloud modality. By default we save the ckpt (optional in the script).

3. Evaluate a saved policy or use it for inference. Please set For example:

   bash复制

   bash scripts/eval_policy.sh dp3 adroit_hammer 0112 0 0

   This will evaluate the saved DP3 policy you just trained. Note: the evaluation script is only provided for deployment/inference. For benchmarking, please use the results logged in wandb during training.

Real Robot

Hardware Setup

1. Franka Robot
2. Allegro Hand
3. L515 Realsense Camera (Note: using the RealSense D435 camera might lead to failure of DP3 due to the very low quality of point clouds)
4. Mounted connection base [link] (connect Franka with Allegro hand)
5. Mounted finger tip [link]

Software

1. Ubuntu 20.04.01 (tested)
2. Franka Interface Control
3. Frankx (High-Level Motion Library for the Franka Emika Robot)
4. Allegro Hand Controller - Noetic

Every collected real robot demonstration (episode length: T) is a dictionary:

1. "point_cloud": Array of shape (T, Np, 6), Np is the number of point clouds, 6 denotes [x, y, z, r, g, b]. Note: it is highly suggested to crop out the table/background and only leave the useful point clouds in your observation, which demonstrates effectiveness in our real-world experiments.
2. "image": Array of shape (T, H, W, 3)
3. "depth": Array of shape (T, H, W)
4. "agent_pos": Array of shape (T, Nd), Nd is the action dim of the robot agent, i.e. 22 for our dexhand tasks (6d position of end effector + 16d joint position)
5. "action": Array of shape (T, Nd). We use relative end-effector position control for the robot arm and relative joint-angle position control for the dex hand.

For training and evaluation, you should process the point clouds (cropping using a bounding box and FPS downsampling) as described in the paper. We also provide an example script (here).

You can try using our provided real world data to train the policy.

1. Download the real robot data. Put the data under 3D-Diffusion-Policy/data/ folder, e.g. 3D-Diffusion-Policy/data/realdex_drill.zarr, please keep the path the same as 'zarr_path' in the task's yaml file.
2. Train the policy. For example:

  bash scripts/train_policy.sh dp3 realdex_drill 0112 0 0

1. For real-world deployment code, you might refer to iDP3 for a reference.

Visualizer

We provide a simple visualizer to visualize point clouds for the convenience of debugging in headless machines. You could install it by

cd visualizer
pip install -e .

Then you could visualize point clouds by

import visualizer
your_pointcloud = ... # your point cloud data, numpy array with shape (N, 3) or (N, 6)
visualizer.visualize_pointcloud(your_pointcloud)

This will show the point cloud in a web browser