百科.dev
全部条目在线工具AI 编程趋势榜开源项目技术资讯提交条目
登录
< 返回工具列表
D

DreamMat

> 编程语言
开源

[SIGGRAPH2024] DreamMat: 利用基于几何和光线的扩散模型生成高质量的 PBR 材质

397 stars0 点赞1 次浏览
访问官网GitHub

工具介绍

[SIGGRAPH2024] DreamMat: 利用基于几何和光线的扩散模型生成高质量的 PBR 材质

DreamMat

DreamMat: High-quality PBR Material Generation with Geometry- and Light-aware Diffusion Models

Paper | Project page

Preparation for inference

  1. Install packages in requirements.txt. We test our model on 3090/4090/V100/A6000 with 11.8 CUDA and 2.0.0 pytorch.

    bash
    git clone https://github.com/zzzyuqing/DreamMat.git
    cd DreamMat
    pip install -r requirements.txt
  2. Install Blender

    Download blender-3.2.2-linux-x64.tar.xz

    Run:

    bash
    tar -xvf blender-3.2.2-linux-x64.tar.xz
    export PATH=$PATH:path_to_blender/blender-3.2.2-linux-x64
  3. Download the pre-trained ControlNet checkpoints here or from hugging face, and put it to the threestudio_dreammat/model/controlnet

  4. A docker env can be found at https://hub.docker.com/repository/docker/zzzyuqing/dreammat_image/general

Inference

bash
cd threestudio_dreammat
sh cmd/run_examples.sh

Upon initial execution, each model will undergo pre-rendering using Blender, with an approximate duration of 15 minutes on a 4090 GPU. During this period, there will be no output; thus, patience is requested. For subsequent runs, the blender_generate can be set to false to bypass this process.

Geometry- and Light-aware ControlNet

You can also train your own geometry- and light-aware ControlNet. The methods for dataset generation and the training code are presented as follows.

Preparation for training

Make sure the environment map folder structure as

bash
dataset
|-- 
    |-- map1
        |-- map1.exr
    |-- map2
        |-- map2.exr
    |-- map3
        |-- map3.exr
    |-- map4
        |-- map4.exr
    |-- map5
        |-- map5.exr

Run the following code to generate pre-rendered data for training

bash
cd controlnet_train
blender -b -P blender_script_geometry.py -- \
    --object_path ./dataset/model/046e3307c74746a58ec4bea5b33b7b97.glb \
    --output_dir ./dataset/training_data \
    --elevation 30 \
    --num_images 16

blender -b -P blender_script_light.py -- \
    --object_path ./dataset/model/046e3307c74746a58ec4bea5b33b7b97.glb \
    --env_dir ./dataset/envmap \
    --output_dir ./dataset/training_data \
    --elevation 30 \
    --num_images 16

The dataset folder structure will be as follows

bash
dataset
|-- training_data
    |-- 
        |-- color
            |-- 000_color_env1.png
            |-- ...
        |-- depth
            |-- 000.png
            |-- ...
        |-- light
            |-- 000_m0.0r0.0_env1.png
            |-- ...
        |-- normal
            |-- 000.png
            |-- ...
    |-- 
    |-- ...
    

Training ControlNet

before training, make sure that the json file of prompts is in the format of

json
{
    "" : "",
    "" : "",
    "" : "",
    ...
}

and the directory of training data is in the structure of

bash
training_data
|-- 
|-- 
|-- 
|-- ...

We provide several data as examples here.

run the training

bash
cd controlnet_train
accelerate launch diffusers_train_controlnet.py --config config.json 

Acknowledgement

We have intensively borrow codes from the following repositories. Many thanks to the authors for sharing their codes.

  • threestudio
  • stable diffusion
  • CSD
  • NeRO
  • Fantasia3D
  • SyncDreamer
  • diffusers
  • ControlNet

In addition to the 3D model from Objaverse, we express our profound appreciation to the contributors of the following 3D models:

  • Bobcat machine by mohamed ouartassi.
  • Molino De Viento _ Windmill by BC-X.
  • MedivalHouse | house for living | MedivalVilage by JFred-chill.
  • Houseleek plant by matousekfoto.
  • Jagernaut (Beyond Human) by skartemka.
  • Grabfigur by noe-3d.at.
  • Teenage Mutant Ninja Turtles - Raphael by Hellbruch.
  • Cat with jet pack by Muru.
  • Transformers Universe: Autobot Showdown by Primus03.
  • PigMan by Grigorii Ischenko.
  • Bulky Knight by Arthur Krut.
  • Sir Frog by Adrian Carter.
  • Infantry Helmet by Masonsmith2020.
  • Sailing Ship Model by Andrea Spognetta (Spogna).
  • Venice Mask by DailyArt.
  • Bouddha Statue Photoscanned by amcgi.
  • Bunny by vivienne0716.
  • Baby Animals Statuettes by Andrei Alexandrescu.
  • Durian The King of Fruits by Laithai.
  • Wooden Shisa (Okinawan Guardian Lion) by Vlad Erium.

We express our profound gratitude to Ziyi Yang for his insightful discussions during the project, and to Lei Yang for his comprehensive coordination and planning. This research work was supported by Information Technology Center and Tencent Lightspeed Studios. Concurrently, we are also exploring more advanced 3D representation and inverse rendering technologies such as Spec-Gaussian and SIRe-IR.

Citation

If you find this repository useful in your project, please cite the following work. :)

…

Issues· 19 开放

查看全部 Issues在 GitHub 打开

暂无开放 Issues,或尚未同步最近议题。

> 标签

Python

暂无评论,来聊聊你的看法吧

> 工具信息

发布日期2026年8月1日
最后更新2026年9月17日
分类编程语言
定价开源

> 相关工具

T
TypeScript
JavaScript 的超集,为前端与全栈提供静态类型
P
Python
通用编程语言,广泛用于 Web、数据与 AI
G
Go
Google 推出的简洁高效系统语言