百科.dev
全部条目在线工具AI 编程趋势榜开源项目技术资讯提交条目
登录
< 返回工具列表
D

diffusers-torchao

> 编程语言
开源

使用 torchao 和 diffusers (推理和 FP8 训练)优化扩散模型的端到端配方。

399 stars0 点赞2 次浏览
访问官网GitHub

工具介绍

使用 torchao 和 diffusers (推理和 FP8 训练)优化扩散模型的端到端配方。

diffusers-torchao

**Optimize image and video generation with diffusers, torchao, combining torch.compile() **

We provide end-to-end inference and experimental training recipes to use torchao with diffusers in this repo. We demonstrate 53.88% speedup on Flux.1-Dev* and 27.33% speedup on CogVideoX-5b when comparing compiled quantized models against their standard bf16 counterparts**.

*The experiments were run on a single H100, 80 GB GPU. **The experiments were run on a single A100, 80 GB GPU. For a single H100, the speedup is 33.04%

Updates

  • torchao is now officially supported as a quantization backend in diffusers. Check out the docs for more details.
  • torchao is being integrated into diffusers as an official quantization backend. Be on the lookout for this PR to get merged.
  • torchao will soon be added as a quantization backend in diffusers, making it even easier to use with diffusers.
  • Check out our new AoT compilation and serialization guide to reduce framework overheads.

No-frills code:

diff
from diffusers import FluxPipeline
+ from torchao.quantization import autoquant
import torch 

pipeline = FluxPipeline.from_pretrained(
    "black-forest-labs/FLUX.1-dev", torch_dtype=torch.bfloat16
).to("cuda")
+ pipeline.transformer = autoquant(pipeline.transformer, error_on_unseen=False)
image = pipeline(
    "a dog surfing on moon", guidance_scale=3.5, num_inference_steps=50
).images[0]

Throw in torch.compile() to make it go brrr:

diff
# If you are using "autoquant" then you should compile first and then
# apply autoquant.
+ pipeline.transformer.to(memory_format=torch.channels_last)
+ pipeline.transformer = torch.compile(
+    pipeline.transformer, mode="max-autotune", fullgraph=True
+)

This, alone, is sufficient to cut down inference time for Flux.1-Dev from 6.431 seconds to 3.483 seconds on an H100. Check out the inference directory for the code.

[!NOTE] Quantizing to a supported datatype and using base precision as fp16 can lead to overflows. The recommended base precision for CogVideoX-2b is fp16 while that of CogVideoX-5b is bf16. If comparisons were to be made in fp16, the speedup gains would be ~23% and ~32% respectively.

Table of contents

  • Environment
  • Benchmarking results
  • Reducing quantization time and peak memory
  • Training with FP8
  • Serialization and loading quantized models
  • Things to keep in mind when benchmarking
  • Benefitting from torch.compile()

Environment

We conducted all our experiments on a single A100 (80GB) and H100 GPUs. Since we wanted to benefit from torch.compile(), we used relatively modern cards here. For older cards, same memory savings (demonstrated more below) can be obtained.

We always default to using the PyTorch nightly, updated diffusers and torchao codebases. We used CUDA 12.2.

Benchmarking results

We benchmark two models (Flux.1-Dev and CogVideoX) using different supported quantization datatypes in torchao. The results are as follows:

Flux.1 Dev Benchmarks

Additional Results

ckpt_id batch_size fuse compile compile_vae quantization sparsify model_memory inference_memory time
black-forest-labs/FLUX.1-dev 4 True True False fp8wo False 22.368 35.616 16.204
black-forest-labs/FLUX.1-dev 8 False False False None False 31.438 47.509 49.438
black-forest-labs/FLUX.1-dev 8 False True False None False 31.439 47.506 31.685
black-forest-labs/FLUX.1-dev 1 False True False int8dq False 20.386 31.608 3.406
black-forest-labs/FLUX.1-dev 4 False True False int8wo False 20.387 31.609 16.08
black-forest-labs/FLUX.1-dev 8 False True False fp8dq False 20.357 36.425 23.393
black-forest-labs/FLUX.1-dev 8 True True False int8dq False 22.397 38.464 24.696
black-forest-labs/FLUX.1-dev 8 False False False int8dq False 20.386 36.458 333.567
black-forest-labs/FLUX.1-dev 4 True False False fp8dq False 22.361 35.826 26.259
black-forest-labs/FLUX.1-dev 8 False True False int8dq False 20.386 36.453 24.725
black-forest-labs/FLUX.1-dev 1 True True False int8wo False 22.396 35.616 4.574
black-forest-labs/FLUX.1-dev 1 False True False fp8wo False 20.363 31.607 4.395
black-forest-labs/FLUX.1-dev 8 True False False int8wo False 22.397 38.468 57.274
black-forest-labs/FLUX.1-dev 4 True False False int8dq False 22.396 35.616 219.687
black-forest-labs/FLUX.1-dev 4 False False False None False 31.438 39.49 24.828
black-forest-labs/FLUX.1-dev 1 True True False fp8dq False 22.363 35.827 3.192
black-forest-labs/FLUX.1-dev 1 False False False fp8dq False 20.356 31.817 8.622
black-forest-labs/FLUX.1-dev 8 False False False fp8dq False 20.357 36.428 55.097
black-forest-labs/FLUX.1-dev 4 False False False int8wo False 20.384 31.606 29.414
black-forest-labs/FLUX.1-dev 1 True False False fp8wo False 22.371 35.618 8.33
black-forest-labs/FLUX.1-dev 1 False False False int8dq False 20.386 31.608 130.498
black-forest-labs/FLUX.1-dev 8 True True False fp8wo False 22.369 38.436 31.718
black-forest-labs/FLUX.1-dev 4 False False False fp8wo False 20.363 31.607 26.61
black-forest-labs/FLUX.1-dev 1 True False False int8wo False 22.397 35.616 8.49
black-forest-labs/FLUX.1-dev 8 True False False fp8dq False 22.363 38.433 51.547
black-forest-labs/FLUX.1-dev 4 False True False fp8dq False 20.359 31.82 11.919
black-forest-labs/FLUX.1-dev 4 False True False None False 31.438 39.488 15.948
black-forest-labs/FLUX.1-dev 4 True True False int8dq False 22.397 35.616 12.594
black-forest-labs/FLUX.1-dev 1 True True False fp8wo False 22.369 35.616 4.326
black-forest-labs/FLUX.1-dev 4 True False False int8wo False 22.397 35.617 29.394
black-forest-labs/FLUX.1-dev 1 False False False fp8wo False 20.362 31.607 8.402
black-forest-labs/FLUX.1-dev 8 True False False int8dq False 22.397 38.468 322.688
black-forest-labs/FLUX.1-dev 1 False False False int8wo False 20.385 31.607 8.551
black-forest-labs/FLUX.1-dev 8 True True False fp8dq False 22.363 38.43 23.261
black-forest-labs/FLUX.1-dev 4 False False False fp8dq False 20.356 31.817 28.154
black-forest-labs/FLUX.1-dev 1 True False False int8dq False 22.397 35.616 119.736
black-forest-labs/FLUX.1-dev 8 True False False fp8wo False 22.369 38.441 51.311
black-forest-labs/FLUX.1-dev 4 False True False fp8wo False 20.363 31.607 16.232
black-forest-labs/FLUX.1-dev 4 True True False int8wo False 22.399 35.619 16.158
black-forest-labs/FLUX.1-dev 8 False False False fp8wo False 20.363 36.434 51.223
black-forest-labs/FLUX.1-dev 4 False False False int8dq False 20.385 31.607 221.588
black-forest-labs/FLUX.1-dev 1 True False False fp8dq False 22.364 35.829 7.34
black-forest-labs/FLUX.1-dev 1 False False False None False 31.438 33.851 6.573
black-forest-labs/FLUX.1-dev 4 True True False fp8dq False 22.363 35.827 11.885
black-forest-labs/FLUX.1-dev 1 False True False int8wo False 20.384 31.606 4.615
black-forest-labs/FLUX.1-dev 8 False True False int8wo False 20.386 36.453 31.159
black-forest-labs/FLUX.1-dev 1 True True False int8dq False 22.397 35.617 3.357
black-forest-labs/FLUX.1-dev 1 False True False fp8dq False 20.357 31.818 3.243
black-forest-labs/FLUX.1-dev 4 False True Fal

Issues· 4 开放

查看全部 Issues在 GitHub 打开

暂无开放 Issues,或尚未同步最近议题。

> 标签

Pythonarchitecture-optimizationdiffusion-modelsfluxtext-to-image

暂无评论,来聊聊你的看法吧

> 工具信息

发布日期2026年8月1日
最后更新2026年9月17日
分类编程语言
定价开源

> 相关工具

T
TypeScript
JavaScript 的超集,为前端与全栈提供静态类型
P
Python
通用编程语言,广泛用于 Web、数据与 AI
G
Go
Google 推出的简洁高效系统语言