End-to-end recipes for optimizing diffusion models with torchao and diffusers (inference and FP8 training).
End-to-end recipes for optimizing diffusion models with torchao and diffusers (inference and FP8 training).
**Optimize image and video generation with diffusers, torchao, combining torch.compile() **
We provide end-to-end inference and experimental training recipes to use torchao with diffusers in this repo. We demonstrate 53.88% speedup on Flux.1-Dev* and 27.33% speedup on CogVideoX-5b when comparing compiled quantized models against their standard bf16 counterparts**.
*The experiments were run on a single H100, 80 GB GPU. **The experiments were run on a single A100, 80 GB GPU. For a single H100, the speedup is 33.04%
torchao is now officially supported as a quantization backend in diffusers. Check out the docs for more details. torchao is being integrated into diffusers as an official quantization backend. Be on the lookout for this PR to get merged. torchao will soon be added as a quantization backend in diffusers, making it even easier to use with diffusers. No-frills code:
from diffusers import FluxPipeline
+ from torchao.quantization import autoquant
import torch
pipeline = FluxPipeline.from_pretrained(
"black-forest-labs/FLUX.1-dev", torch_dtype=torch.bfloat16
).to("cuda")
+ pipeline.transformer = autoquant(pipeline.transformer, error_on_unseen=False)
image = pipeline(
"a dog surfing on moon", guidance_scale=3.5, num_inference_steps=50
).images[0]Throw in torch.compile() to make it go brrr:
# If you are using "autoquant" then you should compile first and then
# apply autoquant.
+ pipeline.transformer.to(memory_format=torch.channels_last)
+ pipeline.transformer = torch.compile(
+ pipeline.transformer, mode="max-autotune", fullgraph=True
+)This, alone, is sufficient to cut down inference time for Flux.1-Dev from 6.431 seconds to 3.483 seconds on an H100. Check out the inference directory for the code.
[!NOTE] Quantizing to a supported datatype and using base precision as fp16 can lead to overflows. The recommended base precision for CogVideoX-2b is fp16 while that of CogVideoX-5b is bf16. If comparisons were to be made in fp16, the speedup gains would be ~23% and ~32% respectively.
Table of contents
torch.compile()We conducted all our experiments on a single A100 (80GB) and H100 GPUs. Since we wanted to benefit from torch.compile(), we used relatively modern cards here. For older cards, same memory savings (demonstrated more below) can be obtained.
We always default to using the PyTorch nightly, updated diffusers and torchao codebases. We used CUDA 12.2.
We benchmark two models (Flux.1-Dev and CogVideoX) using different supported quantization datatypes in torchao. The results are as follows:
Additional Results
| ckpt_id | batch_size | fuse | compile | compile_vae | quantization | sparsify | model_memory | inference_memory | time |
|---|---|---|---|---|---|---|---|---|---|
| black-forest-labs/FLUX.1-dev | 4 | True | True | False | fp8wo | False | 22.368 | 35.616 | 16.204 |
| black-forest-labs/FLUX.1-dev | 8 | False | False | False | None | False | 31.438 | 47.509 | 49.438 |
| black-forest-labs/FLUX.1-dev | 8 | False | True | False | None | False | 31.439 | 47.506 | 31.685 |
| black-forest-labs/FLUX.1-dev | 1 | False | True | False | int8dq | False | 20.386 | 31.608 | 3.406 |
| black-forest-labs/FLUX.1-dev | 4 | False | True | False | int8wo | False | 20.387 | 31.609 | 16.08 |
| black-forest-labs/FLUX.1-dev | 8 | False | True | False | fp8dq | False | 20.357 | 36.425 | 23.393 |
| black-forest-labs/FLUX.1-dev | 8 | True | True | False | int8dq | False | 22.397 | 38.464 | 24.696 |
| black-forest-labs/FLUX.1-dev | 8 | False | False | False | int8dq | False | 20.386 | 36.458 | 333.567 |
| black-forest-labs/FLUX.1-dev | 4 | True | False | False | fp8dq | False | 22.361 | 35.826 | 26.259 |
| black-forest-labs/FLUX.1-dev | 8 | False | True | False | int8dq | False | 20.386 | 36.453 | 24.725 |
| black-forest-labs/FLUX.1-dev | 1 | True | True | False | int8wo | False | 22.396 | 35.616 | 4.574 |
| black-forest-labs/FLUX.1-dev | 1 | False | True | False | fp8wo | False | 20.363 | 31.607 | 4.395 |
| black-forest-labs/FLUX.1-dev | 8 | True | False | False | int8wo | False | 22.397 | 38.468 | 57.274 |
| black-forest-labs/FLUX.1-dev | 4 | True | False | False | int8dq | False | 22.396 | 35.616 | 219.687 |
| black-forest-labs/FLUX.1-dev | 4 | False | False | False | None | False | 31.438 | 39.49 | 24.828 |
| black-forest-labs/FLUX.1-dev | 1 | True | True | False | fp8dq | False | 22.363 | 35.827 | 3.192 |
| black-forest-labs/FLUX.1-dev | 1 | False | False | False | fp8dq | False | 20.356 | 31.817 | 8.622 |
| black-forest-labs/FLUX.1-dev | 8 | False | False | False | fp8dq | False | 20.357 | 36.428 | 55.097 |
| black-forest-labs/FLUX.1-dev | 4 | False | False | False | int8wo | False | 20.384 | 31.606 | 29.414 |
| black-forest-labs/FLUX.1-dev | 1 | True | False | False | fp8wo | False | 22.371 | 35.618 | 8.33 |
| black-forest-labs/FLUX.1-dev | 1 | False | False | False | int8dq | False | 20.386 | 31.608 | 130.498 |
| black-forest-labs/FLUX.1-dev | 8 | True | True | False | fp8wo | False | 22.369 | 38.436 | 31.718 |
| black-forest-labs/FLUX.1-dev | 4 | False | False | False | fp8wo | False | 20.363 | 31.607 | 26.61 |
| black-forest-labs/FLUX.1-dev | 1 | True | False | False | int8wo | False | 22.397 | 35.616 | 8.49 |
| black-forest-labs/FLUX.1-dev | 8 | True | False | False | fp8dq | False | 22.363 | 38.433 | 51.547 |
| black-forest-labs/FLUX.1-dev | 4 | False | True | False | fp8dq | False | 20.359 | 31.82 | 11.919 |
| black-forest-labs/FLUX.1-dev | 4 | False | True | False | None | False | 31.438 | 39.488 | 15.948 |
| black-forest-labs/FLUX.1-dev | 4 | True | True | False | int8dq | False | 22.397 | 35.616 | 12.594 |
| black-forest-labs/FLUX.1-dev | 1 | True | True | False | fp8wo | False | 22.369 | 35.616 | 4.326 |
| black-forest-labs/FLUX.1-dev | 4 | True | False | False | int8wo | False | 22.397 | 35.617 | 29.394 |
| black-forest-labs/FLUX.1-dev | 1 | False | False | False | fp8wo | False | 20.362 | 31.607 | 8.402 |
| black-forest-labs/FLUX.1-dev | 8 | True | False | False | int8dq | False | 22.397 | 38.468 | 322.688 |
| black-forest-labs/FLUX.1-dev | 1 | False | False | False | int8wo | False | 20.385 | 31.607 | 8.551 |
| black-forest-labs/FLUX.1-dev | 8 | True | True | False | fp8dq | False | 22.363 | 38.43 | 23.261 |
| black-forest-labs/FLUX.1-dev | 4 | False | False | False | fp8dq | False | 20.356 | 31.817 | 28.154 |
| black-forest-labs/FLUX.1-dev | 1 | True | False | False | int8dq | False | 22.397 | 35.616 | 119.736 |
| black-forest-labs/FLUX.1-dev | 8 | True | False | False | fp8wo | False | 22.369 | 38.441 | 51.311 |
| black-forest-labs/FLUX.1-dev | 4 | False | True | False | fp8wo | False | 20.363 | 31.607 | 16.232 |
| black-forest-labs/FLUX.1-dev | 4 | True | True | False | int8wo | False | 22.399 | 35.619 | 16.158 |
| black-forest-labs/FLUX.1-dev | 8 | False | False | False | fp8wo | False | 20.363 | 36.434 | 51.223 |
| black-forest-labs/FLUX.1-dev | 4 | False | False | False | int8dq | False | 20.385 | 31.607 | 221.588 |
| black-forest-labs/FLUX.1-dev | 1 | True | False | False | fp8dq | False | 22.364 | 35.829 | 7.34 |
| black-forest-labs/FLUX.1-dev | 1 | False | False | False | None | False | 31.438 | 33.851 | 6.573 |
| black-forest-labs/FLUX.1-dev | 4 | True | True | False | fp8dq | False | 22.363 | 35.827 | 11.885 |
| black-forest-labs/FLUX.1-dev | 1 | False | True | False | int8wo | False | 20.384 | 31.606 | 4.615 |
| black-forest-labs/FLUX.1-dev | 8 | False | True | False | int8wo | False | 20.386 | 36.453 | 31.159 |
| black-forest-labs/FLUX.1-dev | 1 | True | True | False | int8dq | False | 22.397 | 35.617 | 3.357 |
| black-forest-labs/FLUX.1-dev | 1 | False | True | False | fp8dq | False | 20.357 | 31.818 | 3.243 |
| black-forest-labs/FLUX.1-dev | 4 | False | True | Fal |
No open issues yet, or sync has not completed.