百科.dev
全部条目在线工具AI 编程趋势榜开源项目技术资讯提交条目
登录
< 返回工具列表
S

STream3R

> 编程语言
开源

使用因果 Transformer 的动态 3D 基础模型。 [ICLR 2026]

393 stars0 点赞2 次浏览
访问官网GitHub

工具介绍

使用因果 Transformer 的动态 3D 基础模型。 [ICLR 2026]

Abstract

:fire: News

  • [May 8, 2026] Check out our new 4D Reconstruction Model 4RC, ICML 2026.
  • [Mar 9, 2026] Check out the DUSt3R-based metric-scale STream3R[α] version on the alpha branch.
  • [Jan 26, 2026] Accepted to ICLR 2026!
  • [Sep 16, 2025] The complete training code is released!
  • [Aug 22, 2025] The evaluation code is now available!
  • [Aug 15, 2025] Our inference code and weights are released!

Installation

  1. Clone Repo

    bash
    git clone https://github.com/NIRVANALAN/STream3R
    cd STream3R
  2. Create Conda Environment

    bash
    conda create -n stream3r python=3.11 cmake=3.14.0 -y
    conda activate stream3r
  3. Install Python Dependencies

    Important: Install Torch based on your CUDA version. For example, for Torch 2.8.0 + CUDA 12.6:

    bash
    # Install Torch
    pip3 install torch torchvision --index-url https://download.pytorch.org/whl/cu126
    
    # Install other dependencies
    pip install -r requirements.txt
    
    # Install STream3R as a package
    pip install -e .

:computer: Inference

You can now try STream3R with the following code. The checkpoint will be downloaded automatically from Hugging Face.

You can set the inference mode to causal for causal attention, window for sliding window attention (with a default window size of 5), or full for bidirectional attention.

…

A script is already at inference_stream3r.py.

We also support a KV cache version to enable streaming input using StreamSession. The StreamSession takes sequential input and processes them one by one, making it suitable for real-time or low-latency applications. This streaming 3D reconstruction pipeline can be applied in various scenarios such as real-time robotics, autonomous navigation, online 3D understanding and SLAM. An example usage is shown below:

…

:zap: Demo

You can run the demo built on VGG-T's code using the script app.py with the following command:

bash
python app.py

Code Structure

The repository is structured as follows:

STream3R/
├── stream3r/                    
│   ├── models/                  
│   │   ├── stream3r.py            
│   │   ├── multiview_dust3r_module.py  
│   │   └── components/               
│   ├── dust3r/                 
│   ├── croco/                  
│   ├── utils/                  
│   └── stream_session.py          
├── configs/                     
├── examples/                    
├── assets/                      
├── app.py                          
├── requirements.txt                 
├── setup.py                        
└── README.md                       

:100: Quantitive Results

3D Reconstruction Comparison on NRGBD.

Method Type Acc Mean ↓ Acc Med. ↓ Comp Mean ↓ Comp Med. ↓ NC Mean ↑ NC Med. ↑
VGG-T FA 0.073 0.018 0.077 0.021 0.910 0.990
DUSt3R Optim 0.144 0.019 0.154 0.018 0.870 0.982
MASt3R Optim 0.085 0.033 0.063 0.028 0.794 0.928
MonST3R Optim 0.272 0.114 0.287 0.110 0.758 0.843
Spann3R Stream 0.416 0.323 0.417 0.285 0.684 0.789
CUT3R Stream 0.099 0.031 0.076 0.026 0.837 0.971
StreamVGGT Stream 0.084 0.044 0.074 0.041 0.861 0.986
Ours Stream 0.057 0.014 0.028 0.013 0.910 0.993

Read our full paper for more insights.

⏳ GPU Memory Usage and Runtime

We report the peak GPU memory usage (VRAM) and runtime of our full model for processing each streaming input using the StreamSession implementation. All experiments were conducted at a common resolution of 518 × 384 on a single H200 GPU. The benchmark includes both Causal for causal attention and Window for sliding window attention with a window size of 5.

Run Time (s).

Num of Frames 1 20 40 80 100 120 140 180 200
Causal 0.1164 0.2034 0.3060 0.4986 0.5945 0.6947 0.7916 0.9911 1.1703
Window 0.1167 0.1528 0.1523 0.1517 0.1515 0.1512 0.1482 0.1443 0.1463

VRAM (GB).

Num of Frames 1 20 40 80 100 120 140 180 200
Causal 5.49 9.02 12.92 21.00 25.03 29.10 33.21 41.31 45.41
Window 5.49 6.53 6.53 6.53 6.53 6.53 6.53 6.53 6.53

:hotsprings: Training

  1. Prepare Dataset

    We follow CUT3R to preprocess the dataset for training.

  2. Set Up Config

    Update training config file configs/experiment/stream3r/stream3r.yaml as needed. For example:

    • Set pretrained to the path of the VGG-T checkpoint.
    • Set data_root to the directory where you saved the processed dataset.
  3. Launch training with:

    bash
    python stream3r/train.py experiment=stream3r/stream3r
  4. After training, you can convert the checkpoint into a state_dict file, for example: Click to view code for checkpoint conversion

…

Evaluation

The evaluation follows MonST3R and Spann3R, CUT3R.

  1. Prepare Evaluation Dataset

    We follow the dataset preparation guides from MonST3R and Spann3R to prepare the datasets. For convenience, we provide the processed datasets on Hugging Face, which can be downloaded directly.

    The datasets should be organized as follows under the root directiory of the project:

    data/
    ├── 7scenes
    ├── bonn
    ├── kitti
    ├── neural_rgbd
    ├── nyu-v2
    ├── scannetv2
    ├── sintel
    └── tum
  2. Run Evaluation

    Use the provided scripts to evaluate different tasks.

    For Video Depth and Camera Pose Estimation, some datasets contain more than 100 images. To reduce memory usage, we use StreamSession to process frames sequentially while managing the KV cache.

    Monodepth

    bash
    bash eval/monodepth/run.sh

    Results will be saved in eval_results/monodepth/${model_name}/${data}/metric.json.

    Video Depth

    bash
    bash eval/video_depth/run.sh

    Results will be saved in eval_results/video_depth/${model_name}/${data}/result_scale.json.

    Camera Pose Estimation

    bash
    bash eval/relpose/run.sh

    Results will be saved in eval_results/relpose/${model_name}/${data}/_error_log.txt.

    Multi-view Reconstruction

    bash
    bash eval/mv_recon/run.sh

    Results will be saved in eval_results/mv_recon/${model_name}/${data}/logs_all.txt.

:calendar: TODO

  • Release evaluation code.
  • Release training code.
  • Release the metric-scale version, see branch alpha.

:page_with_curl: License

This project is licensed under NTU S-Lab License 1.0. Redistribution and use should follow this license.

:pencil: Citation

If you find our code or paper helps, please consider citing bib:

bibtex
@inproceedings{stream3r2025,
title={{STream3R}: Scalable Sequential {3D} Reconstruction with Causal Transformer},
author={Lan, Yushi and Luo, Yihang and Hong, Fangzhou and Zhou, Shangchen and Chen, Honghua and Lyu, Zhaoyang and Yang, Shuai and Dai, Bo and Loy, Chen Change and Pan, Xingang},
booktitle = {ICLR},
year={2026}
}

:pencil: Acknowledgments

We recognize several concurrent works on streaming methods. We encourage you to check them out:

StreamVGGT  |  CUT3R  |  SLAM3R  |  Spann3R

STream3R is built on the shoulders of several outstanding open-source projects. Many thanks to the following exceptional projects:

VGG-T  |  Fast3R  |  DUSt3R  |  MonST3R  |  Viser

:mailbox: Contact

If you have any question, please feel free to contact us via [email protected] or Github issues.

Issues· 0 开放

查看全部 Issues在 GitHub 打开

暂无开放 Issues,或尚未同步最近议题。

> 标签

Python

暂无评论,来聊聊你的看法吧

> 工具信息

发布日期2026年8月1日
最后更新2026年9月17日
分类编程语言
定价开源

> 相关工具

T
TypeScript
JavaScript 的超集,为前端与全栈提供静态类型
P
Python
通用编程语言,广泛用于 Web、数据与 AI
G
Go
Google 推出的简洁高效系统语言