CUT3R

Continuous 3D Perception Model with Persistent State

QianqianWang*, Yifei Zhang*, Aleksander Holynski, Alexei A Efros, Angjoo Kanazawa

(*: equal contribution)

Table of Contents

• TODO
• Get Started
  • Installation
  • Checkpoints
  • Inference
• Datasets
• Evaluation
  • Datasets
  • Evaluation Scripts
• Training and Fine-tuning
• Acknowledgements
• Citation

TODO

• Release multi-view stereo results of DL3DV dataset.
• Online demo integrated with WebCam

Getting Started

Installation

1. Clone CUT3R.

git clone https://github.com/CUT3R/CUT3R.git
cd CUT3R

1. Create the environment.

conda create -n cut3r python=3.11 cmake=3.14.0
conda activate cut3r
conda install pytorch torchvision pytorch-cuda=12.1 -c pytorch -c nvidia  # use the correct version of cuda for your system
pip install -r requirements.txt
# issues with pytorch dataloader, see https://github.com/pytorch/pytorch/issues/99625
conda install 'llvm-openmp<16'
# for training logging
pip install git+https://github.com/nerfstudio-project/gsplat.git
# for evaluation
pip install evo
pip install open3d

1. Compile the cuda kernels for RoPE (as in CroCo v2).

cd src/croco/models/curope/
python setup.py build_ext --inplace
cd ../../../../

Download Checkpoints

We currently provide checkpoints on Google Drive:

cut3r_224_linear_4.pth is our intermediate checkpoint and cut3r_512_dpt_4_64.pth is our final checkpoint.

To download the weights, run the following commands:

cd src
# for 224 linear ckpt
gdown --fuzzy https://drive.google.com/file/d/11dAgFkWHpaOHsR6iuitlB_v4NFFBrWjy/view?usp=drive_link 
# for 512 dpt ckpt
gdown --fuzzy https://drive.google.com/file/d/1Asz-ZB3FfpzZYwunhQvNPZEUA8XUNAYD/view?usp=drive_link
cd ..

Inference

To run the inference code, you can use the following command:

…

Output results will be saved to output_dir.

Currently, we accelerate the feedforward process by processing inputs in parallel within the encoder, which results in linear memory consumption as the number of frames increases.

Datasets

Our training data includes 32 datasets listed below. We provide processing scripts for all of them. Please download the datasets from their official sources, and refer to preprocess.md for processing scripts and more information about the datasets.

• ARKitScenes
• BlendedMVS
• CO3Dv2
• MegaDepth
• ScanNet++
• ScanNet
• WayMo Open dataset
• WildRGB-D
• Map-free
• TartanAir
• UnrealStereo4K
• Virtual KITTI 2
• 3D Ken Burns
• BEDLAM
• COP3D
• DL3DV
• Dynamic Replica
• EDEN
• Hypersim
• IRS
• Matterport3D
• MVImgNet
• MVS-Synth
• OmniObject3D
• PointOdyssey
• RealEstate10K
• SmartPortraits
• Spring
• Synscapes
• UASOL
• UrbanSyn
• HOI4D

Evaluation

Datasets

Please follow MonST3R and Spann3R to prepare Sintel, Bonn, KITTI, NYU-v2, TUM-dynamics, ScanNet, 7scenes and Neural-RGBD datasets.

The datasets should be organized as follows:

data/
├── 7scenes
├── bonn
├── kitti
├── neural_rgbd
├── nyu-v2
├── scannetv2
├── sintel
└── tum

Evaluation Scripts

Please refer to the eval.md for more details.

Training and Fine-tuning

Please refer to the train.md for more details.

Acknowledgements

Our code is based on the following awesome repositories:

• DUSt3R
• MonST3R
• Spann3R
• Viser

We thank the authors for releasing their code!

Citation

If you find our work useful, please cite:

@article{wang2025continuous,
  title={Continuous 3D Perception Model with Persistent State},
  author={Wang, Qianqian and Zhang, Yifei and Holynski, Aleksander and Efros, Alexei A and Kanazawa, Angjoo},
  journal={arXiv preprint arXiv:2501.12387},
  year={2025}
}