Baike.dev
All toolsOnline toolsAI codingTrendingOpen sourceNewsSubmit
Log in
< Back to tools
D

Diffusion_models_from_scratch

> 编程语言
Open source

Creating a diffusion model from scratch in PyTorch to learn exactly how they work.

396 stars0 likes0 views
WebsiteGitHub

About

Creating a diffusion model from scratch in PyTorch to learn exactly how they work.

Update!

This repo is quite old with respect to diffusion models. I wanted to update it to a flow-based model that using a DiT architecture instead of a DDPM-based model using a U-net architecture. A more modern version of this repo can be found here

I also have a repo training stable diffusion 3 from scratch at this repo!

Summary

This repo is composed of DDPM, DDIM, and Classifier-Free guided models trained on ImageNet 64x64. More information can be found below.

To go along with this repo, I also wrote an article explaining the algorithms behind it.

Contents

  • Current Additions
  • Environment Setup
  • Downloading Pre-Trained Models
  • Downloading Training Data
  • Directory Structure
  • Train A Model
  • Generate Images With Pretrained Models
  • Calculating FID for a pretrained model
  • My Results
  • References

Current Additions

This repo has the following Diffusion features:

  • Normal DDPM
  • Improved DDPM with cosine scheduler and variance prediction
  • DDIM for faster inference
  • Classifier-Free Guidance to improve image quality

Instead of going into each of the parts here, you can read an article I wrote which explains each part in detail.

Environment Setup

First, download the data from this repo using the following on the command line

bash
git clone https://github.com/gmongaras/Diffusion_models_from_scratch.git
cd Diffusion_models_from_scratch/

(Optional) If you don't want to change your environment, you can first create a virtual environment:

bash
pip install virtualenv
python -m venv MyEnv/

Activate the virtual environment: https://docs.python.org/3/library/venv.html#how-venvs-work

Windows: MyEnv\Scripts\activate.bat

Linux: source MyEnv/bin/activate

Before running any scripts, make sure to download the correct packages and package versions. You can do so by running the following commands to upgrade pip and install the necessary package versions:

bash
pip install pip -U
pip install -U requirements.txt

Note: PyTorch should be installed with cuda enabled if training and probably should have cuda if generating images, but is not required. The cuda version downloaded may be different from the one needed. The cuda versions and how to download them can be found below:

https://pytorch.org/get-started/locally/

Now the enviroment should be setup properly.

Downloading Pre-Trained Models

Pre-Trained Model Notes

I have several pre-trained models available to download or varying model architecture types. There are 5 model types based on the u-net block construction.

  • res ➜ conv ➜ clsAtn ➜ chnAtn (Res-Conv)
  • res ➜ clsAtn ➜ chnAtn (Res)
  • res ➜ res ➜ clsAtn ➜ chnAtn (Res-Res)
  • res ➜ res ➜ clsAtn ➜ atn ➜ chnAtn (Res-Res-Atn)
  • res ➜ clsAtn ➜ chnAtn with 192 channels (Res Large)

The above notation comes from the Train A Model section under the blk_types parameter.

Each model was trained with the following parameters unless otherwise specified:

  1. Image Resolution: 64x64
  2. Initial embedding channel: 128
  3. Channel multiplier — 1
  4. Number of U-net blocks — 3
  5. Timesteps — 1000
  6. VLB weighting Lambda — 0.001
  7. Beta Scheduler — Cosine
  8. Batch Size — 128 (across 8 GPUs, so 1024)
  9. Gradient Accumulation Steps — 1
  10. Number of steps (Note: This is not epochs, a step is a single gradient update to the model)— 600,000
  11. Learning Rate — 3*10^-4 = 0.0003
  12. Time embedding dimension size— 512
  13. Class embedding dimension size — 512
  14. Probability of null class for classifier-free guidance — 0.2
  15. Attention resolution — 16

Below are some training notes:

  • I used 8 parallel GPUs each with a batch size of 128. SO, the batch size is 8*128 = 1024.
  • I trained each model for a total of 600,000 steps. Note that this isn't number of epochs, but rather model updates. The models look to be able to be trained for longer since the FID values look the be decreasing even at 600,000 steps if you wish to continue training from a pre-trained checkpoint.
  • Training the smaller models (res-conv, res, res-res) took 6-7 days to train and the larger models took about 8 days to train on 8 A100s.

Picking a Model

To pick a model, I suggest looking at the results. The lower the FID score, the better better the outputs of the model are. The best models according to the results are:

  • res-res-atn:
    • 358 epochs (358e)
    • 450000 steps (450000s)
    • The files for this model are:
      • model file: model_358e_450000s.pkl
      • model file: model_params_358e_450000s.json
      • optim file: optim_358e_450000s.pkl
  • res-res:
    • 438 epochs (438e)
    • 550000 steps (550000s)
    • The files for this model are:
      • model file: model_438e_550000s.pkl
      • model metadata: model_params_438e_550000s.json
      • optimizer: optim_438e_550000s.pkl

Downloading A Model

Once the model has been picked, you can download a model at the following link:

Google Drive link

For training from a checkpoint you need to download three files for a model:

  • The model .pkl file (ex: model_438e_550000s.pkl)
  • The model metadata .json file (ex: model_params_438e_550000s.json)
  • The optimizer .pkl file (ex: optim_438e_550000s.pkl)

For inference/generation you only need to download two files for the model:

  • The model .pkl file (ex: model_438e_550000s.pkl)
  • The model metadata .json file (ex: model_params_438e_550000s.json)

Put these files in the models/ directory to easily load them in when training/generating.

Downloading Training Data

Imagenet data can be downloaded from the following link: https://image-net.org/download-images.php

To get the data, you must first request access and be accepted to download the Imagenet data. I trained my models on Imagenet 64x64

Once downloaded, you should pur both the Imagenet64_train_part1.zip and Imagenet64_train_part2.zip in the data/ directory.

The zip files are in the correct directory, run the following script to load the data into the necessary format:

python data/loadImagenet64.py

If you wish to load the data into memory before training, run the script below. Otherwise, the data will be extracted from disk as needed.

python data/make_massive_tensor.py

The directory should look as follows when all data is downloaded: Directory Structure

Directory Structure

If you download both pretrained models and the training data, your directory should look like the following tree.

…

Train A Model

Before training a model, make sure you setup the environment and downloaded the data

After the above is complete, you can run the training script as follows from the root directory of this repo:

torchrun --nproc_per_node=[num_gpus] src/train.py --[params]

  • [num_gpus] is replaced by the number of desired GPUs to parallelize training on
  • [params] is replaced with any of the parameters listed below

For example:

torchrun --nproc_per_node=8 src/train.py --blk_types res,res,clsAtn,chnAtn --batchSize 32

The above example runs the code with the following parameters:

  • Run on 8 parallel GPUs.
  • Each u-net block is composed of res->res->clsAtn->chnAtn sequential blocks.
  • batchSize is 32 on each GPU, so a total batch size of 8*32 = 256.

torchrun --nproc_per_node=1 src/train.py --loadModel True --loadDir models/models_res --loadFile model_479e_600000s.pkl --optimFile optim_479e_600000s.pkl --loadDefFile model_params_479e_600000s.json --gradAccSteps 2

The above example loads in a pre-trained model for checkpoint:

  • Use 1 GPU
  • Load the model file model_479e_600000s.pkl
  • Load the optimizer file optim_479e_600000s.pkl
  • Load the model metadata file model_params_479e_600000s.json
  • Use 2 gradient accumulation steps

The parameters of the script are as follows:

Data Parameters

  • inCh [3] - Number of input channels for the input data.
  • data_path [data/Imagenet64] - Path to the ImageNet 64x64 dataset.
  • load_into_mem [True] - True to load all ImageNet data into memory, False to load the data from disk as needed.

Model Parameters

  • embCh [128] - Number of channels in the top layer of the U-net. Note, this is scaled by 2^(chMult*layer) each U-net layer
  • chMult [1] - At each U-net layer, at what scale should the channels be multiplied by? Each layer has embCh2^(chMultlayer) channels
  • num_layers [3] - Number of U-net layers. A value of 3 has a depth of 3, meaning there are 3 down layers and 3 up layers in the U-net
  • blk_types [res,clsAtn,chnAtn] - How should the residual block be structured? (list of res, conv, clsAtn, atn, and/or chnAtn.
    • res: Normal residual block.
    • conv: ConvNext block.
    • clsAtn: Attention block to include extra class information (Q, K are cls features).
    • atn: Attention ViT block for the hidden features.
    • clsAtn: Efficient-lightweight attention block over the feature channels.
    • EX: res,res,conv,clsAtn,chnAtn
  • T [1000] - Number of timesteps in the diffusion process
  • beta_sched [cosine] - Noise scheduler to use in the diffusion process. Can be either linear or cosine
  • t_dim [512] - Dimension of the vector encoding for the time information.
  • c_dim [512] - Dimension of the vector encoding for the class information. NOTE: Use -1 for no class information
  • atn_resolution [16] - Resolution of the attention block (atn). The resolution splits the image into patches of that resolution to act as vectors. Ex: a resolution of 16 creates 16x16 patches and flattens them as feature vectors.

Training Parameters

  • Lambda [0.001] - Weighting term between the variance and mean loss in the model.
  • batchSize [128] - Batch size on a single GPU. If using multiple GPUs, this batch size will be multiplied by the GPU count.
  • gradAccSteps [1] - Number of steps to breakup the batchSize into. Instead of taking 1 massive step where the whole batch is loaded into memory, the batchSize is broken up into sizes of batchSize//gradAccSteps so that it can fit into memory. Mathematically, the update will be the same, as a single batch update, but the update is distributed across smaller updates to fit into memory. A higher value takes more time, but uses less memory.
  • device [gpu] - Device to put the model on. Use "gpu" to put it on one or multiple Cuda devices or "cpu" to put it on the CPU. CPU should only be used for testing.
  • epochs [1000000] - Number of epochs to train for.
  • lr [0.0003] - Model learning rate.
  • p_uncond [0.2] - Probability of training on a null class for classifier-free guidance. Note that good values are 0.1 or 0.2. (only used if c_dim is not None)
  • use_importance [False] - True to use importance sampling for values of t, False to use uniform sampling.

Saving Parameters

  • saveDir [models/] - Directory to save models checkpoints to. NOTE that three files will be saved: the model .pkl file, the model metadata .json file, and the optimizer .pkl file for training reloading
  • numSaveSteps [10000] -"Number of steps until a new model checkpoint is saved. This is not the number of epochs, rather it's the number of time the model has updates. NOTE that three files will be saved: the model .pkl file, the model metadata .json file, and the optimizer .pkl file for training reloading.

Model

Issues· 3 open

View all issuesOpen on GitHub

No open issues yet, or sync has not completed.

> Tags

Pythonclassifier-free-guidanceddimddpmdiffusion-models

No comments yet. Be the first to share.

> Details

PublishedAug 1, 2026
UpdatedSep 17, 2026
Category编程语言
PricingOpen source

> Related tools

T
TypeScript
JavaScript 的超集,为前端与全栈提供静态类型
P
Python
通用编程语言,广泛用于 Web、数据与 AI
G
Go
Google 推出的简洁高效系统语言