splatt3r

Open source

Official repository for Splatt3R: Zero-shot Gaussian Splatting from Uncalibrated Image Pairs

798 stars0 likes0 views

About

Official repository for Splatt3R: Zero-shot Gaussian Splatting from Uncalibrated Image Pairs

Official implementation of Splatt3R: Zero-shot Gaussian Splatting from Uncalibrated Image Pairs, a feed-forward model that can directly predict 3D Gaussians from uncalibrated images.

News

[2024/08/27] We release the initial version of the codebase, the paper, the project webpage, and the Gradio demo!!

Installation

Clone Splatt3R

bash

git clone https://github.com/btsmart/splatt3r.git
cd splatt3r

Setup Anaconda Environment

bash

conda env create -f environment.yml
pip install git+https://github.com/dcharatan/diff-gaussian-rasterization-modified

(Optional) Compile the CUDA kernels for RoPE (as in MASt3R and CroCo v2)

bash

cd src/mast3r_src/dust3r/croco/models/curope/
python setup.py build_ext --inplace
cd ../../../../../../

Checkpoints

We train our model using the pretrained MASt3R_ViTLarge_BaseDecoder_512_catmlpdpt_metric checkpoint from the MASt3R authors, available from the MASt3R GitHub repo. This checkpoint is placed at the file path checkpoints/MASt3R_ViTLarge_BaseDecoder_512_catmlpdpt_metric.pth.

A pretrained Splatt3R model can be downloaded here.

Data

We use ScanNet++ to train our model. We download the data from the official ScanNet++ homepage and process the data using SplaTAM's modified version of the ScanNet++ toolkit. We save the processed data to the 'processed' subfolder of the ScanNet++ root directory.

Our generated test coverage files, and our training and testing splits, can be downloaded here, and placed in data/scannetpp.

Demo

The Gradio demo can be run using python demo.py, which loads our trained checkpoint from Hugging Face.

This demo generates a .ply file that represents the scene, which can be downloaded and rendered using online 3D Gaussian Splatting viewers such as here or here.

Our example images and .ply files are available for download here.

Training

Our training run can be recreated by running python main.py configs/main.yaml. Other configurations, such as those for the ablations, can be found in the configs folder.

BibTeX

If you find Splatt3R useful for your research and applications, please cite us using this BibTex:

@article{smart2024splatt3r,
      title={Splatt3R: Zero-shot Gaussian Splatting from Uncalibrated Image Pairs}, 
      author={Brandon Smart and Chuanxia Zheng and Iro Laina and Victor Adrian Prisacariu},
      year={2024},
      eprint={2408.13912},
      archivePrefix={arXiv},
      primaryClass={cs.CV},
      url={https://arxiv.org/abs/2408.13912}, 
}