This repo is quite old with respect to diffusion models. I wanted to update it to a flow-based model that using a DiT architecture instead of a DDPM-based model using a U-net architecture. A more modern version of this repo can be found here
I also have a repo training stable diffusion 3 from scratch at this repo!
Summary
This repo is composed of DDPM, DDIM, and Classifier-Free guided models trained on ImageNet 64x64. More information can be found below.
To go along with this repo, I also wrote an article explaining the algorithms behind it.
Before running any scripts, make sure to download the correct packages and package versions. You can do so by running the following commands to upgrade pip and install the necessary package versions:
Note: PyTorch should be installed with cuda enabled if training and probably should have cuda if generating images, but is not required. The cuda version downloaded may be different from the one needed. The cuda versions and how to download them can be found below:
I have several pre-trained models available to download or varying model architecture types. There are 5 model types based on the u-net block construction.
res ➜ conv ➜ clsAtn ➜ chnAtn (Res-Conv)
res ➜ clsAtn ➜ chnAtn (Res)
res ➜ res ➜ clsAtn ➜ chnAtn (Res-Res)
res ➜ res ➜ clsAtn ➜ atn ➜ chnAtn (Res-Res-Atn)
res ➜ clsAtn ➜ chnAtn with 192 channels (Res Large)
The above notation comes from the Train A Model section under the blk_types parameter.
Each model was trained with the following parameters unless otherwise specified:
Image Resolution: 64x64
Initial embedding channel: 128
Channel multiplier — 1
Number of U-net blocks — 3
Timesteps — 1000
VLB weighting Lambda — 0.001
Beta Scheduler — Cosine
Batch Size — 128 (across 8 GPUs, so 1024)
Gradient Accumulation Steps — 1
Number of steps (Note: This is not epochs, a step is a single gradient update to the model)— 600,000
Learning Rate — 3*10^-4 = 0.0003
Time embedding dimension size— 512
Class embedding dimension size — 512
Probability of null class for classifier-free guidance — 0.2
Attention resolution — 16
Below are some training notes:
I used 8 parallel GPUs each with a batch size of 128. SO, the batch size is 8*128 = 1024.
I trained each model for a total of 600,000 steps. Note that this isn't number of epochs, but rather model updates. The models look to be able to be trained for longer since the FID values look the be decreasing even at 600,000 steps if you wish to continue training from a pre-trained checkpoint.
Training the smaller models (res-conv, res, res-res) took 6-7 days to train and the larger models took about 8 days to train on 8 A100s.
Picking a Model
To pick a model, I suggest looking at the results. The lower the FID score, the better better the outputs of the model are. The best models according to the results are:
res-res-atn:
358 epochs (358e)
450000 steps (450000s)
The files for this model are:
model file: model_358e_450000s.pkl
model file: model_params_358e_450000s.json
optim file: optim_358e_450000s.pkl
res-res:
438 epochs (438e)
550000 steps (550000s)
The files for this model are:
model file: model_438e_550000s.pkl
model metadata: model_params_438e_550000s.json
optimizer: optim_438e_550000s.pkl
Downloading A Model
Once the model has been picked, you can download a model at the following link:
The above example loads in a pre-trained model for checkpoint:
Use 1 GPU
Load the model file model_479e_600000s.pkl
Load the optimizer file optim_479e_600000s.pkl
Load the model metadata file model_params_479e_600000s.json
Use 2 gradient accumulation steps
The parameters of the script are as follows:
Data Parameters
inCh [3] - Number of input channels for the input data.
data_path [data/Imagenet64] - Path to the ImageNet 64x64 dataset.
load_into_mem [True] - True to load all ImageNet data into memory, False to load the data from disk as needed.
Model Parameters
embCh [128] - Number of channels in the top layer of the U-net. Note, this is scaled by 2^(chMult*layer) each U-net layer
chMult [1] - At each U-net layer, at what scale should the channels be multiplied by? Each layer has embCh2^(chMultlayer) channels
num_layers [3] - Number of U-net layers. A value of 3 has a depth of 3, meaning there are 3 down layers and 3 up layers in the U-net
blk_types [res,clsAtn,chnAtn] - How should the residual block be structured? (list of res, conv, clsAtn, atn, and/or chnAtn.
res: Normal residual block.
conv: ConvNext block.
clsAtn: Attention block to include extra class information (Q, K are cls features).
atn: Attention ViT block for the hidden features.
clsAtn: Efficient-lightweight attention block over the feature channels.
EX: res,res,conv,clsAtn,chnAtn
T [1000] - Number of timesteps in the diffusion process
beta_sched [cosine] - Noise scheduler to use in the diffusion process. Can be either linear or cosine
t_dim [512] - Dimension of the vector encoding for the time information.
c_dim [512] - Dimension of the vector encoding for the class information. NOTE: Use -1 for no class information
atn_resolution [16] - Resolution of the attention block (atn). The resolution splits the image into patches of that resolution to act as vectors. Ex: a resolution of 16 creates 16x16 patches and flattens them as feature vectors.
Training Parameters
Lambda [0.001] - Weighting term between the variance and mean loss in the model.
batchSize [128] - Batch size on a single GPU. If using multiple GPUs, this batch size will be multiplied by the GPU count.
gradAccSteps [1] - Number of steps to breakup the batchSize into. Instead of taking 1 massive step where the whole batch is loaded into memory, the batchSize is broken up into sizes of batchSize//gradAccSteps so that it can fit into memory. Mathematically, the update will be the same, as a single batch update, but the update is distributed across smaller updates to fit into memory. A higher value takes more time, but uses less memory.
device [gpu] - Device to put the model on. Use "gpu" to put it on one or multiple Cuda devices or "cpu" to put it on the CPU. CPU should only be used for testing.
epochs [1000000] - Number of epochs to train for.
lr [0.0003] - Model learning rate.
p_uncond [0.2] - Probability of training on a null class for classifier-free guidance. Note that good values are 0.1 or 0.2. (only used if c_dim is not None)
use_importance [False] - True to use importance sampling for values of t, False to use uniform sampling.
Saving Parameters
saveDir [models/] - Directory to save models checkpoints to. NOTE that three files will be saved: the model .pkl file, the model metadata .json file, and the optimizer .pkl file for training reloading
numSaveSteps [10000] -"Number of steps until a new model checkpoint is saved. This is not the number of epochs, rather it's the number of time the model has updates. NOTE that three files will be saved: the model .pkl file, the model metadata .json file, and the optimizer .pkl file for training reloading.