SuGaR

## Abstract _We propose a method to allow precise and extremely fast mesh extraction from 3D Gaussian Splatting (SIGGRAPH 2023). Gaussian Splatting has recently become very popular as it yields realistic rendering while being significantly faster to train than NeRFs. It is however challenging to extract a mesh from the millions of tiny 3D Gaussians as these Gaussians tend to be unorganized after optimization and no method has been proposed so far. Our first key contribution is a regularization term that encourages the 3D Gaussians to align well with the surface of the scene. We then introduce a method that exploits this alignment to sample points on the real surface of the scene and extract a mesh from the Gaussians using Poisson reconstruction, which is fast, scalable, and preserves details, in contrast to the Marching Cubes algorithm usually applied to extract meshes from Neural SDFs. Finally, we introduce an optional refinement strategy that binds Gaussians to the surface of the mesh, and jointly optimizes these Gaussians and the mesh through Gaussian splatting rendering. This enables easy editing, sculpting, rigging, animating, or relighting of the Gaussians using traditional softwares (Blender, Unity, Unreal Engine, etc.) by manipulating the mesh instead of the Gaussians themselves. Retrieving such an editable mesh for realistic rendering is done within minutes with our method, compared to hours with the state-of-the-art method on neural SDFs, while providing a better rendering quality in terms of PSNR, SSIM and LPIPS._ ## BibTeX ``` @article{guedon2023sugar, title={SuGaR: Surface-Aligned Gaussian Splatting for Efficient 3D Mesh Reconstruction and High-Quality Mesh Rendering}, author={Gu{\'e}don, Antoine and Lepetit, Vincent}, journal={CVPR}, year={2024} } ``` ## Updates and To-do list Updates

• [09/18/2024] Improved the quality of the extracted meshes with the new `dn_consistency` regularization method, and added compatibility with the new Blender add-on for composition and animation.
• [01/09/2024] Added a dedicated, real-time viewer to let users visualize and navigate in the reconstructed scenes (hybrid representation, textured mesh and wireframe mesh).
• [12/20/2023] Added a short notebook showing how to render images with the hybrid representation using the Gaussian Splatting rasterizer.
• [12/18/2023] Code release.

To-do list

• Viewer: Add option to load the postprocessed mesh.
• Mesh extraction: Add the possibility to edit the extent of the background bounding box.
• Tips&Tricks: Add to the README.md file (and the webpage) some tips and tricks for using SuGaR on your own data and obtain better reconstructions (see the tips provided by user kitmallet).
• Improvement: Add an if block to sugar_extractors/coarse_mesh.py to skip foreground mesh reconstruction and avoid triggering an error if no surface point is detected inside the foreground bounding box. This can be useful for users that want to reconstruct "background scenes".
• Using precomputed masks with SuGaR: Add a mask functionality to the SuGaR optimization, to allow the user to mask out some pixels in the training images (like white backgrounds in synthetic datasets).
• Using SuGaR with Windows: Adapt the code to make it compatible with Windows. Due to path-writing conventions, the current code is not compatible with Windows.
• Synthetic datasets: Add the possibility to use the NeRF synthetic dataset (which has a different format than COLMAP scenes)
• Composition and animation: Finish to clean the code for composition and animation, and add it to the sugar_scene/sugar_compositor.py script.
• Composition and animation: Make a tutorial on how to use the scripts in the blender directory and the sugar_scene/sugar_compositor.py class to import composition and animation data into PyTorch and apply it to the SuGaR hybrid representation.

## Overview As we explain in the paper, SuGaR optimization starts with first optimizing a 3D Gaussian Splatting model for 7k iterations with no additional regularization term. Consequently, the current implementation contains a version of the original 3D Gaussian Splatting code, and we built our model as a wrapper of a vanilla 3D Gaussian Splatting model. Please note that, even though this wrapper implementation is convenient for many reasons, it may not be the most optimal one for memory usage. The full SuGaR pipeline consists of 4 main steps, and an optional one: 1. **Short vanilla 3DGS optimization**: optimizing a vanilla 3D Gaussian Splatting model for 7k iterations, in order to let Gaussians position themselves in the scene. 2. **SuGaR optimization**: optimizing Gaussians alignment with the surface of the scene. 3. **Mesh extraction**: extracting a mesh from the optimized Gaussians. 4. **SuGaR refinement**: refining the Gaussians and the mesh together to build a hybrid Mesh+Gaussians representation. 5. **Textured mesh extraction (Optional)**: extracting a traditional textured mesh from the refined SuGaR model as a tool for visualization, composition and animation in Blender using our Blender add-on. We provide a dedicated script for each of these steps, as well as a script `train_full_pipeline.py` that runs the entire pipeline. We explain how to use this script in the next sections.

Please note that the final step, _Textured mesh extraction_, is optional but is enabled by default in the `train_full_pipeline.py` script. Indeed, it is very convenient to have a traditional textured mesh for visualization, composition and animation using traditional softwares such as Blender. If you installed Nvdiffrast as described below, this step should only take a few seconds anyway.
Below is another example of a scene showing a robot with a black and specular material. The following images display the hybrid representation (Mesh + Gaussians on the surface), the mesh with a traditional colored UV texture, and a depth map of the mesh: ## Installation Click here to see content. ### 0. Requirements The software requirements are the following: - Conda (recommended for easy setup) - C++ Compiler for PyTorch extensions - CUDA toolkit 11.8 for PyTorch extensions - C++ Compiler and CUDA SDK must be compatible Please refer to the original 3D Gaussian Splatting repository for more details about requirements. ### 1. Clone the repository Start by cloning this repository: ```shell # HTTPS git clone https://github.com/Anttwo/SuGaR.git --recursive ``` or ```shell # SSH git clone [email protected]:Anttwo/SuGaR.git --recursive ``` ### 2. Creating the Conda environment To create and activate the Conda environment with all the required packages, go inside the `SuGaR/` directory and run the following command: ```shell python install.py conda activate sugar ``` This script will automatically create a Conda environment named `sugar` and install all the required packages. It will also automatically install the 3D Gaussian Splatting rasterizer as well as the Nvdiffrast library for faster mesh rasterization. If you encounter any issues with the installation, you can try to follow the detailed instructions below to install the required packages manually. Detailed instructions for manual installation #### a) Install the required Python packages To install the required Python packages and activate the environment, go inside the `SuGaR/` directory and run the following commands: ```shell conda env create -f environment.yml conda activate sugar ``` If this command fails to create a working environment, you can try to install the required packages manually by running the following commands: ```shell conda create --name sugar -y python=3.9 conda activate sugar conda install pytorch==2.0.1 torchvision==0.15.2 torchaudio==2.0.2 pytorch-cuda=11.8 -c pytorch -c nvidia conda install -c fvcore -c iopath -c conda-forge fvcore iopath conda install pytorch3d==0.7.4 -c pytorch3d conda install -c plotly plotly conda install -c conda-forge rich conda install -c conda-forge plyfile==0.8.1 conda install -c conda-forge jupyterlab conda install -c conda-forge nodejs conda install -c conda-forge ipywidgets pip install open3d pip install --upgrade PyMCubes ``` #### b) Install the Gaussian Splatting rasterizer Run the following commands inside the `SuGaR` directory to install the additional Python submodules required for Gaussian Splatting: ```shell cd gaussian_splatting/submodules/diff-gaussian-rasterization/ pip install -e . cd ../simple-knn/ pip install -e . cd ../../../ ``` Please refer to the 3D Gaussian Splatting repository for more details. #### c) (Optional) Install Nvdiffrast for faster Mesh Rasterization Installing Nvdiffrast is optional but will greatly speed up the textured mesh extraction step, from a few minutes to less than 10 seconds. ```shell git clone https://github.com/NVlabs/nvdiffrast cd nvdiffrast pip install . cd ../ ``` ## Quick Start Click here to see content. ### Training from scratch You can run the following single script to optimize a full SuGaR model from scratch using a COLMAP dataset: ```shell python train_full_pipeline.py -s -r <"dn_consistency", "density" or "sdf"> --high_poly True --export_obj True ``` You can choose the regularization method with the `-r` argument, which can be `"dn_consistency"`, `"density"` or `"sdf"`. We recommend using the newer `"dn_consistency"` regularization for best quality meshes, but the results presented in the paper were obtained with the `"density"` regularization for object-centered scenes and `"sdf"` for scenes with a challenging background, such as the Mip-NeRF 360 dataset. You can also replace the `--high_poly True` argument with `--low_poly True` to extract a mesh with 200k vertices instead of 1M, and 6 Gaussians per triangle instead of 1. Moreover, you can add `--refinement_time "short"`, `"medium"` or `"long"` to set the time spent on the refinement step. The default is `"long"` (15k iterations), but `"short"` (2k iterations) can be enough to produce a good-looking hybrid representation. Finally, you can choose to export a traditional textured mesh with the `--export_obj` argument. This step is optional but is enabled by default in the `train_full_pipeline.py` script, as the mesh is required for using the Blender add-on and editing, combining or animating scenes in Blender. Results are saved in the `output/` directory. Please click here to see the most important arguments for the `train_full_pipeline.py` script. | Parameter | Type | Description | | :-------: | :--: | :---------: | | `--scene_path` / `-s`