Dynamic 3D Foundation Model using Causal Transformer. [ICLR 2026]
Dynamic 3D Foundation Model using Causal Transformer. [ICLR 2026]
Abstract
Clone Repo
git clone https://github.com/NIRVANALAN/STream3R
cd STream3RCreate Conda Environment
conda create -n stream3r python=3.11 cmake=3.14.0 -y
conda activate stream3rInstall Python Dependencies
Important: Install Torch based on your CUDA version. For example, for Torch 2.8.0 + CUDA 12.6:
# Install Torch
pip3 install torch torchvision --index-url https://download.pytorch.org/whl/cu126
# Install other dependencies
pip install -r requirements.txt
# Install STream3R as a package
pip install -e .You can now try STream3R with the following code. The checkpoint will be downloaded automatically from Hugging Face.
You can set the inference mode to causal for causal attention, window for sliding window attention (with a default window size of 5), or full for bidirectional attention.
…A script is already at inference_stream3r.py.
We also support a KV cache version to enable streaming input using StreamSession. The StreamSession takes sequential input and processes them one by one, making it suitable for real-time or low-latency applications. This streaming 3D reconstruction pipeline can be applied in various scenarios such as real-time robotics, autonomous navigation, online 3D understanding and SLAM. An example usage is shown below:
…You can run the demo built on VGG-T's code using the script app.py with the following command:
python app.pyThe repository is structured as follows:
STream3R/
├── stream3r/
│ ├── models/
│ │ ├── stream3r.py
│ │ ├── multiview_dust3r_module.py
│ │ └── components/
│ ├── dust3r/
│ ├── croco/
│ ├── utils/
│ └── stream_session.py
├── configs/
├── examples/
├── assets/
├── app.py
├── requirements.txt
├── setup.py
└── README.md 3D Reconstruction Comparison on NRGBD.
| Method | Type | Acc Mean ↓ | Acc Med. ↓ | Comp Mean ↓ | Comp Med. ↓ | NC Mean ↑ | NC Med. ↑ |
|---|---|---|---|---|---|---|---|
| VGG-T | FA | 0.073 | 0.018 | 0.077 | 0.021 | 0.910 | 0.990 |
| DUSt3R | Optim | 0.144 | 0.019 | 0.154 | 0.018 | 0.870 | 0.982 |
| MASt3R | Optim | 0.085 | 0.033 | 0.063 | 0.028 | 0.794 | 0.928 |
| MonST3R | Optim | 0.272 | 0.114 | 0.287 | 0.110 | 0.758 | 0.843 |
| Spann3R | Stream | 0.416 | 0.323 | 0.417 | 0.285 | 0.684 | 0.789 |
| CUT3R | Stream | 0.099 | 0.031 | 0.076 | 0.026 | 0.837 | 0.971 |
| StreamVGGT | Stream | 0.084 | 0.044 | 0.074 | 0.041 | 0.861 | 0.986 |
| Ours | Stream | 0.057 | 0.014 | 0.028 | 0.013 | 0.910 | 0.993 |
Read our full paper for more insights.
We report the peak GPU memory usage (VRAM) and runtime of our full model for processing each streaming input using the StreamSession implementation. All experiments were conducted at a common resolution of 518 × 384 on a single H200 GPU. The benchmark includes both Causal for causal attention and Window for sliding window attention with a window size of 5.
Run Time (s).
| Num of Frames | 1 | 20 | 40 | 80 | 100 | 120 | 140 | 180 | 200 |
|---|---|---|---|---|---|---|---|---|---|
| Causal | 0.1164 | 0.2034 | 0.3060 | 0.4986 | 0.5945 | 0.6947 | 0.7916 | 0.9911 | 1.1703 |
| Window | 0.1167 | 0.1528 | 0.1523 | 0.1517 | 0.1515 | 0.1512 | 0.1482 | 0.1443 | 0.1463 |
VRAM (GB).
| Num of Frames | 1 | 20 | 40 | 80 | 100 | 120 | 140 | 180 | 200 |
|---|---|---|---|---|---|---|---|---|---|
| Causal | 5.49 | 9.02 | 12.92 | 21.00 | 25.03 | 29.10 | 33.21 | 41.31 | 45.41 |
| Window | 5.49 | 6.53 | 6.53 | 6.53 | 6.53 | 6.53 | 6.53 | 6.53 | 6.53 |
Prepare Dataset
We follow CUT3R to preprocess the dataset for training.
Set Up Config
Update training config file configs/experiment/stream3r/stream3r.yaml as needed. For example:
pretrained to the path of the VGG-T checkpoint.data_root to the directory where you saved the processed dataset.Launch training with:
python stream3r/train.py experiment=stream3r/stream3rAfter training, you can convert the checkpoint into a state_dict file, for example:
Click to view code for checkpoint conversion
…The evaluation follows MonST3R and Spann3R, CUT3R.
Prepare Evaluation Dataset
We follow the dataset preparation guides from MonST3R and Spann3R to prepare the datasets. For convenience, we provide the processed datasets on Hugging Face, which can be downloaded directly.
The datasets should be organized as follows under the root directiory of the project:
data/
├── 7scenes
├── bonn
├── kitti
├── neural_rgbd
├── nyu-v2
├── scannetv2
├── sintel
└── tumRun Evaluation
Use the provided scripts to evaluate different tasks.
For Video Depth and Camera Pose Estimation, some datasets contain more than 100 images. To reduce memory usage, we use StreamSession to process frames sequentially while managing the KV cache.
bash eval/monodepth/run.sh Results will be saved in eval_results/monodepth/${model_name}/${data}/metric.json.
bash eval/video_depth/run.sh Results will be saved in eval_results/video_depth/${model_name}/${data}/result_scale.json.
bash eval/relpose/run.sh Results will be saved in eval_results/relpose/${model_name}/${data}/_error_log.txt.
bash eval/mv_recon/run.sh Results will be saved in eval_results/mv_recon/${model_name}/${data}/logs_all.txt.
This project is licensed under NTU S-Lab License 1.0. Redistribution and use should follow this license.
If you find our code or paper helps, please consider citing bib:
@inproceedings{stream3r2025,
title={{STream3R}: Scalable Sequential {3D} Reconstruction with Causal Transformer},
author={Lan, Yushi and Luo, Yihang and Hong, Fangzhou and Zhou, Shangchen and Chen, Honghua and Lyu, Zhaoyang and Yang, Shuai and Dai, Bo and Loy, Chen Change and Pan, Xingang},
booktitle = {ICLR},
year={2026}
}We recognize several concurrent works on streaming methods. We encourage you to check them out:
StreamVGGT | CUT3R | SLAM3R | Spann3R
STream3R is built on the shoulders of several outstanding open-source projects. Many thanks to the following exceptional projects:
VGG-T | Fast3R | DUSt3R | MonST3R | Viser
If you have any question, please feel free to contact us via [email protected] or Github issues.
No open issues yet, or sync has not completed.