hierarchical-3d-gaussians

A Hierarchical 3D Gaussian Representation for Real-Time Rendering of Very Large Datasets

Bernhard Kerbl*, Andreas Meuleman*, Georgios Kopanas, Michael Wimmer, Alexandre Lanvin, George Drettakis (* indicates equal contribution)

Project page | Paper

This repository contains the official authors' implementation associated with the paper "A Hierarchical 3D Gaussian Representation for Real-Time Rendering of Very Large Datasets". We explain the different steps required to run our algorithm. We use a "toy example" of 1500 images organized in 2 chunks to illustrate each step of the method and facilitate reproduction. The full datasets presented in the paper will be released as soon as the data protection process is completed (please stay tuned).

Bibliography:

@Article{hierarchicalgaussians24,
      author       = {Kerbl, Bernhard and Meuleman, Andreas and Kopanas, Georgios and Wimmer, Michael and Lanvin, Alexandre and Drettakis, George},
      title        = {A Hierarchical 3D Gaussian Representation for Real-Time Rendering of Very Large Datasets},
      journal      = {ACM Transactions on Graphics},
      number       = {4},
      volume       = {43},
      month        = {July},
      year         = {2024},
      url          = {https://repo-sam.inria.fr/fungraph/hierarchical-3d-gaussians/}
}

Roadmap

Please note that the code release is currently in alpha. We intend to provide fixes for issues that are experienced by users, due to difficulties with setups and/or environments that we did not test on. The below steps were successfully tested on Windows and Ubuntu 22. We appreciate the documentation of issues by users and will try to address them. Furthermore, there are several points that we will integrate in the coming weeks:

• Datasets: We will add links for large-scale datasets that are currently undergoing auditing.
• Windows binaries: Once we have sufficiently tested them, we will add pre-compiled binaries for the viewers on Windows.
• Direct conversion of legacy 3DGS models: we are testing the conversion of scenes trained with vanilla 3DGS to hierarchical models. Once the quality is assured and we have concluded testing, we will document the necessary steps to do so.
• Streaming from disk: currently, data is streamed on-demand to the GPU, however, the viewed dataset must fit into memory. This can become prohibitive in the hierarchy merger and real-time viewer. We will adapt the code to allow dynamic streaming from disk soon.
• Reduce real-time viewer resource usage: the storage configuration for the real-time viewer is unoptimized, and so is the speed. Users can define a VRAM budget for the scene, but it is not used as efficiently as it could be. We will iterate towards making sure that higher quality settings can be achieved with lower budgets and better framerates. We will try to make the budget so that it effectively limits the total application VRAM, including framebuffer structs.

Setup

Make sure to clone the repo using --recursive:

git clone https://github.com/graphdeco-inria/hierarchical-3d-gaussians.git --recursive
cd hierarchical-3d-gaussians

Prerequisite

We tested on Ubuntu 22.04 and Windows 11 using the following:

• CMake 3.22.1
• gcc/g++ 11.4.0 or Visual Studio 2019
• CUDA (11.8, 12.1 or 12.5)
• COLMAP 3.9.1 (for preprocessing only). Linux: build from source. Windows: add the path to the COLMAP.bat directory to the PATH environment variable.

Python environment for optimization

conda create -n hierarchical_3d_gaussians python=3.12 -y
conda activate hierarchical_3d_gaussians
# Replace cu121 with cu118 if using CUDA 11.x 
pip install torch==2.3.0 torchvision==0.18.0 torchaudio==2.3.0 --index-url https://download.pytorch.org/whl/cu121 
pip install -r requirements.txt

Weights for monocular depth estimation

To enable depth loss, download the model weights of one of these methods:

• Depth Anything V2 (suggested): download from Depth-Anything-V2-Large and place it under submodules/Depth-Anything-V2/checkpoints/.
• DPT (used in the paper): download from dpt_large-midas-2f21e586.pt and place it under submodules/DPT/weights/.

Compiling hierarchy generator and merger

cd submodules/gaussianhierarchy
cmake . -B build -DCMAKE_BUILD_TYPE=Release
cmake --build build -j --config Release
cd ../..

Compiling the real-time viewer

For Ubuntu 22.04, install dependencies:

sudo apt install -y cmake libglew-dev libassimp-dev libboost-all-dev libgtk-3-dev libopencv-dev libglfw3-dev libavdevice-dev libavcodec-dev libeigen3-dev libxxf86vm-dev libembree-dev

Clone the hierarchy viewer and build:

cd SIBR_viewers
git clone https://github.com/graphdeco-inria/hierarchy-viewer.git src/projects/hierarchyviewer
cmake . -B build -DCMAKE_BUILD_TYPE=Release -DBUILD_IBR_HIERARCHYVIEWER=ON -DBUILD_IBR_ULR=OFF -DBUILD_IBR_DATASET_TOOLS=OFF -DBUILD_IBR_GAUSSIANVIEWER=OFF 
cmake --build build -j --target install --config Release

Running the method

Our method has two main stages: Reconstruction, that takes a (usually large) set of images as input and outputs a "merged hierarchy", and Runtime, that displays the full hierarchy in real-time.

Reconstruction has two main steps: 1) Preprocessing the input images and 2) Optimization. We present these in detail next. For each step we have automatic scripts that perform all the required steps, and we also provide details about the individual components.

Dataset

To get started, prepare a dataset or download and extract the toy example. The dataset should have sorted images in a folder per camera in ${DATASET_DIR}/inputs/images/ and optional masks (with .png extension) in ${DATASET_DIR}/inputs/masks/. Masks will be multiplied to the input images and renderings before computing loss.

You can also work from our full scenes. As we provide them calibrated and subdivided, you may skip to Generate monocular depth maps. The datasets:

• SmallCity

In the following, replace ${DATASET_DIR} with the path to your dataset or set DATASET_DIR:

# Bash:
DATASET_DIR=<Path to your dataset>

# PowerShell:
${DATASET_DIR} = "<Path to your dataset>"

To skip the reconstruction and only display scenes, download pretrained hierarchies and scaffolds, place them under ${DATASET_DIR}/output/ and follow the viewer instructions. The pretrained hierarchies:

• SmallCity

1. Preprocessing

As in 3dgs we need calibrated cameras and a point cloud to train our hierarchies on.

1.1 Calibrating the cameras

The first step is to generate a "global colmap". The following command uses COLMAP's hierarchical mapper, rectify images and masks, and align and scale the sparse reconstruction to facilitate subdivision.

python preprocess/generate_colmap.py --project_dir ${DATASET_DIR}

Using calibrated images

If your dataset already has COLMAP (with 2D and 3D SfM points) and rectified images, they should be placed under ${DATASET_DIR}/camera_calibration/rectified. As they still need alignment, run:

python preprocess/auto_reorient.py --input_path ${DATASET_DIR}/camera_calibration/rectified/sparse --output_path ${DATASET_DIR}/camera_calibration/aligned/sparse/0

This step takes ~ 47 minutes on our example dataset using a RTX A6000, more details on each steps of the script here.

1.2 Generate chunks

Once the "global colmap" generated, it should be split into chunks. We also run a per-chunk bundle adjustment as COLMAP's hierarchical mapper is faster but less accurate (if your global colmap is accurate, you can skip this time consuming step with --skip_bundle_adjustment).

python preprocess/generate_chunks.py --project_dir ${DATASET_DIR}

This step takes ~ 95 minutes on our example dataset using a RTX A6000, more details on each steps of the script here.

note that by using --use_slurm you can refine the chunks in parallel, remember to set your slurm parameters in preprocess/prepare_chunks.slurm (gpu, account, etc ...).

1.3 Generate monocular depth maps

In order to use depth regularization when training each chunks, depth maps must be generated for each rectified image. Then, depth scaling parameters needs to be computed as well, these two steps can be done using:

python preprocess/generate_depth.py --project_dir ${DATASET_DIR}

Project structure

Now you should have the following file structure, it is required for the training part:

project
└── camera_calibration
    ├── aligned
    │   └── sparse/0
    │       ├── images.bin
    │       ├── cameras.bin
    │       └── points3D.bin
    ├── chunks
    │   ├── 0_0
    │   └── 0_1
    │   .
    │   .
    │   .
    │   └── m_n
    │       ├── center.txt
    │       ├── extent.txt
    │       └── sparse/0
    │           ├── cameras.bin
    │           ├── images.bin
    │           ├── points3d.bin
    │           └── depth_params.json
    └── rectified
        ├── images
        ├── depths
        └── masks

2. Optimization

The scene training process is divided into five steps; 1) we first train a global, coarse 3D Gaussian splatting scene ("the scaffold"), then 2) train each chunk independently in parallel, 3) build the hierarchy, 4) optimize the hierarchy in each chunk and finally 5) consolidate the chunks to create the final hierarchy.

Make sure that you correctly set up your environment and built the hierarchy merger/creator

The full_train.py script performs all these steps to train a hierarchy from a preprocessed scene. While training, the progress can be visualized with the original 3DGS remote viewer (build instructions).

python scripts/full_train.py --project_dir ${DATASET_DIR}

Command Line Arguments

--colmap_dir

Input aligned colmap.

--images_dir

Path to rectified images.

--depths_dir

Path to rectified depths.

--masks_dir

Path to rectified masks.

--chunks_dir

Path to input chunks folder.

--env_name

Name the conda env you created earlier.

--output_dir

Path to output dir.

--use_slurm

Flag to enable parallel tra