Baike.dev
All toolsOnline toolsAI codingTrendingOpen sourceNewsSubmit
Log in
< Back to tools
G

GSFix3D

> 编程语言
Open source

GSFix3D: Diffusion-Guided Repair of Novel Views in Gaussian Splatting

198 stars0 likes0 views
WebsiteGitHub

About

GSFix3D: Diffusion-Guided Repair of Novel Views in Gaussian Splatting

GSFix3D: Diffusion-Guided Repair of Novel Views in Gaussian Splatting

Jiaxin Wei, Stefan Leutenegger, Simon Schaefer

Paper | Video | Project Page | Model

TL;DR: Remove artifacts and fill holes for novel views in 3DGS scenes.

Abstract: Recent developments in 3D Gaussian Splatting have significantly enhanced novel view synthesis, yet generating high-quality renderings from extreme novel viewpoints or partially observed regions remains challenging. Meanwhile, diffusion models exhibit strong generative capabilities, but their reliance on text prompts and lack of awareness of specific scene information hinder accurate 3D reconstruction tasks. To address these limitations, we introduce GSFix3D, a novel framework that improves the visual fidelity in under-constrained regions by distilling prior knowledge from diffusion models into 3D representations, while preserving consistency with observed scene details. At its core is GSFixer, a latent diffusion model obtained via our customized fine-tuning protocol that can leverage both mesh and 3D Gaussians to adapt pretrained generative models to a variety of environments and artifact types from different reconstruction methods, enabling robust novel view repair for unseen camera poses. Moreover, we propose a random mask augmentation strategy that empowers GSFixer to plausibly inpaint missing regions. Experiments on challenging benchmarks demonstrate that our GSFix3D and GSFixer achieve state-of-the-art performance, requiring only minimal scene-specific fine-tuning on captured data. Real-world test further confirms its resilience to potential pose errors.

News

  • [2025-11-18]: Released the code and data of GSFix3D. More models are coming soon, stay tuned!
  • [2025-11-06]: Our paper has been accepted by International Conference on 3D Vision 2026 (3DV)!
  • [2025-08-20]: Released the paper on arXiv.

Setup

We tested our code on: Ubuntu 22.04 LTS, Python 3.11.14, CUDA 12.1, NVIDIA RTX 4500 Ada.

Clone the repository with --recursive flag

bash
git clone https://github.com/GSFix3D/GSFix3D.git --recursive

Create virtual environment

bash
cd GSFix3D
conda create -n gsfix3d python=3.11
conda activate gsfix3d
pip install -r requirements.txt

Build differentiable Gaussian rasterizer

bash
cd diff-gaussian-rasterization
pip install .

Usage

Training from scratch

You can skip training GSFixer from scratch and directly use our pretrained checkpoints and proceed with fine-tuning in the next step. Or you can download Stable Diffusion v2 checkpoint into ${BASE_CKPT_DIR} and use the provided training script to train it from scratch. We notice that the official Stable Diffusion v2 has been removed from Huggingface, but it may still be accessible through third-party sources or other community-maintained archives. Please refer to here for training data preparation.

Set environment parameters for the data directory

bash
export BASE_DATA_DIR=YOUR_DATA_DIR        # directory to training data
export BASE_CKPT_DIR=YOUR_CHECKPOINT_DIR  # directory to pretrained checkpoint

Adjust hyperparameters in YAML files under config/train_gsfixer to determine whether to train with or without dual input

dual_input: true  # or false to train GSFixer with only gs input

You also need to adjust the checkpoint name in config/model_sdv2.yaml if you are using an unofficial model with a different name

pretrained_path: stable-diffusion-2  # checkpoint name in ${BASE_CKPT_DIR}

Run the training script

bash
python scripts/gsfixer/train.py --config config/train_gsfixer.yaml --output_dir 

Note that only UNet and scheduler will be saved in the --output_dir. You need to reuse other model components from the pretrained checkpoint in BASE_CKPT_DIR.

Fine-tuning on captured data

Now you can fine-tune the base GSFixer model on any captured data. First, you need to organize the captured data as follows (here we take the room1 scene from the Replica dataset for example). Specifically, you need to put rendered images from the 3DGS model and (optionally) mesh, as well as the captured ground truth RGB images, under a finetune folder to construct data pairs for fine-tuning. Please refer to here for downloading the Replica and ScanNet++ datasets. We also provide the rendered images from GSFusion, SplaTAM and RTG-SLAM for Replica and ScanNet++ datasets, which can be downloaded here. Note that novel_views and novel_views.json are the novel view data we created for quantitative evaluation on the Replica dataset. Please refer to our paper for more details on the novel view selection process.

Replica
├── cam_params.json
└── room1
    ├── finetune
    |   ├── rendered_gs
    |   ├── rendered_mesh
    |   └── rgb
    ├── novel_views
    ├── novel_views.json
    ├── rendered_novel_views
    |   ├── gs_image_rtg
    |   ├── gs_image_splatam
    |   ├── gs_image_gsfusion
    |   └── mesh_image
    ├── results
    └── traj.txt

Set environment parameters for the data directory

bash
export BASE_DATA_DIR=CAPTURED_DATA_DIR        # directory to captured data
export BASE_CKPT_DIR=GSFIXER_CHECKPOINT_DIR   # directory to pretrained GSFixer checkpoint

Adjust hyperparameters in YAML files under config/finetune_gsfixer

dual_input: true     # or false to fine-tune GSFixer with only gs input
dir: room1/finetune  # path to fine-tuning data

You also need to adjust the checkpoint name in config/model_gsfixer.yaml if you are using your own GSFixer checkpoint with a different name. Note that the checkpoint in use must be consistent with the dual_input flag set in the YAML file above.

pretrained_path: gsfixer-full # checkpoint name in ${BASE_CKPT_DIR}

Run the fine-tuning script

bash
python scripts/gsfixer/train.py --config config/finetune_gsfixer.yaml --output_dir 

Note that only UNet and scheduler will be saved in the --output_dir. You need to reuse other model components from the pretrained GSFixer checkpoint in BASE_CKPT_DIR.

Inference with GSFixer

Run the inference script to fix 2D novel views using the fine-tuned GSFixer model

bash
python scripts/gsfixer/inference.py --checkpoint  --recon_method_type  --data_type  --data_path  --output_dir   --dual_input --eval

The fine-tuned GSFixer checkpoints for the Replica and ScanNet++ datasets will be available soon on Hugging Face. If you are testing on Replica or ScanNet++ dataset, the `` is the path to the organized dataset detailed above and you can use --eval flag to enable evaluation on repaired novel views obtained from GSFixer; otherwise, you need to provide the path to your captured data. --recon_method_type is used to locate input (rendered) gs images with corresponding file name postfix. You can enable dual-conditioning by using --dual_input flag.

Command Line Arguments for inference.py

--checkpoint

Path that contains the fine-tuned GSFixer model or the Hugging Face model ID.

--recon_method_type

File name postfix to locate input gs images. We provide reconstruction data from gsfusion, splatam and rtg.

--data_type

Currently supported data types are replica and scannetpp.

--data_path

Path to your captured data.

--dual_input

Enable dual-conditioning (mesh+gs).

--eval

Enable evaluation on fixed novel views obtained from GSFixer. Note that this flag is only valid for the Replica dataset or ScanNet++ dataset. For your own data, you need to prepare the ground truth for novel views to use this function.

Lifting to 3D with GSFix3D

We provide a script to lift the results of GSFixer back into the 3DGS model

bash
python scripts/gsfix3d/refine_gs.py -m  --data_type  --data_path  --fixed_image_path  --output_dir  --eval

Please refer to here for the expected file structure of . We provide several example 3DGS maps [here](https://cloud.cvai.cit.tum.de/s/feg33Y8wGMGEC9t) for the Replica and ScanNet++ datasets. If you are testing on Replica or ScanNet++ dataset, the is the path to the organized dataset above and you can use --eval flag to enable evaluation on novel views rendered from the refined 3DGS model; otherwise, you need to provide the path to your captured data and --novel_views the novel view camera poses in json file format. Please refer to the next section for more details about how to prepare your own data.

Command Line Arguments for refine_gs.py

-m

Path that contains the 3DGS model.

--data_type

Currently supported data types are replica and scannetpp.

--data_path

Path to your captured data.

--sparse_kf_list (optional)

A keyframe list to only select keyframes from ground truth data for further refinement of the 3DGS model, which is recommended for reducing optimization time.

--fixed_image_path

Path to fixed novel view images obtained from GSFixer.

--novel_views

JSON file path to novel view camera poses.

--eval

Enable evaluation on novel views rendered from the refined 3DGS model. Note that this flag is only valid for the Replica dataset or ScanNet++ dataset. For your own data, you need to prepare the ground truth for novel views to use this function.

Testing on your own data

Prepare the novel views that you want to repair using our provided script

bash
python scripts/gsfix3d/novel_view_capture.py -m  --data_type  --data_path  --output_dir  --render_mesh

Here we only provide a simple script to simplify the process of novel view selection and rendering from selected novel views. Note that our script requires loading a mesh model for interaction in Open3D. The mesh is also needed if you want to render novel view images from the mesh. You can download several example mesh files here for the Replica and ScanNet++ datasets or run GSFusion to obtain both mesh and 3DGS map simultaneously. You can also use other methods (eliminating the requirement for a mesh model) to obtain novel view data (i.e., poses and rendered images), as long as you follow the same format as ours.

Command Line Arguments for novel_view_capture.py

-m

Path that contains the 3DGS model and mesh. The expected file structure of `` is explained here.

--data_type

Currently supported data types are replica and scannetpp. We highly recommend that you organize your customized data in Replica format so that it can be directly recognized by our provided scripts.

--data_path

Path to your customized data.

--render_mesh

Enable mesh rendering from selected novel views.

Citation

If you find our work useful, please cite us:

bibtex
@inproceedings{wei2026gsfix3d,
  title={Gsfix3d: Diffusion-guided repair of novel views in gaussian splatting},
  author={Wei, Jiaxin and Leutenegger, Stefan and Schaefer, Simon},
  booktitle={2026 International Conference on 3D Vision (3DV)},
  pages={1051--1060},
  year={2026},
  organization={IEEE}
}

License

Copyright (c) 2025, Jiaxin Wei

Important Notice

  • This project includes code licensed under the Apache License Version 2.0.
  • This project includes components licensed under the Gaussian-Splatting-License, which restricts the entire project to non-commercial use only.

Issues· 0 open

View all issuesOpen on GitHub

No open issues yet, or sync has not completed.

> Tags

Python

No comments yet. Be the first to share.

> Details

PublishedAug 1, 2026
UpdatedSep 18, 2026
Category编程语言
PricingOpen source

> Related tools

T
TypeScript
JavaScript 的超集,为前端与全栈提供静态类型
P
Python
通用编程语言,广泛用于 Web、数据与 AI
G
Go
Google 推出的简洁高效系统语言