百科.dev
全部条目在线工具AI 编程趋势榜开源项目技术资讯提交条目
登录
< 返回工具列表
F

flash-fft-conv

> 编程语言
开源

FlashFFTConv: 利用 Tensor 核心对长序列进行高效卷积

357 stars0 点赞1 次浏览
访问官网GitHub

工具介绍

FlashFFTConv: 利用 Tensor 核心对长序列进行高效卷积

FlashFFTConv: Efficient Convolutions for Long Sequences with Tensor Cores

This repository contains the official code for FlashFFTConv, a fast algorithm for computing long depthwise convolutions using the FFT algorithm.

FlashFFTConv computes convolutions up to 7.93 times faster than PyTorch FFT convolutions, with up to 8.21 times less memory usage. FlashFFTConv supports convolution kernel lengths up to 4,194,304.

We also provide a fast kernel for short 1D depthwise convolutions (e.g., where the kernel length is on the order of 3/5), which runs 7 times faster than PyTorch Conv1D. This module is useful for additional speedup for language models like Monarch Mixer, H3, and Hyena.

FlashFFTConv: Efficient Convolutions for Long Sequences with Tensor Cores
Daniel Y. Fu*, Hermann Kumbong*, Eric Nguyen, Christopher Ré
Paper: https://arxiv.org/abs/2311.05908
Blog: https://hazyresearch.stanford.edu/blog/2023-11-13-flashfftconv

Examples and Usage

We've been happy to use FlashFFTConv to support various projects, including Monarch Mixer, Hyena/H3, HyenaDNA, and more. FlashFFTConv is also being used to train various new models that haven't been released yet - we'll be updating this README with pointers as they're publicly announced.

Check out the examples folder for end-to-end examples of how to use FlashFFTConv in your models.

You can also run a standalone CIFAR example to see usage as soon as the package is installed:

bash
python standalone_cifar.py

Installation

Requirements: We recommend using the Nvidia PyTorch docker container. We've tested and developed this library on version 23.05.

  • PyTorch 2.0 Required
  • We have tested with CUDA version 12.1 and toolkit version 12.1
  • We have tested this on A100 and H100, but it should work on any Ampere/Hopper architecture (3090, 4090, etc)

To check your CUDA version:

  • Run nvcc --version and check the version number of your CUDA toolkit. Our Docker ships with version 12.1.
  • Run nvidia-smi to check the version of your CUDA drivers. Our Docker ships with version 12.1.

You can install via pip:

bash
pip install git+https://github.com/HazyResearch/flash-fft-conv.git#subdirectory=csrc/flashfftconv
pip install git+https://github.com/HazyResearch/flash-fft-conv.git

Or from source:

bash
git clone https://github.com/HazyResearch/flash-fft-conv.git

cd flash-fft-conv

cd csrc/flashfftconv
python setup.py install

cd ../..

python setup.py install

Once it's installed, you should be able to run the test suite:

pytest -s -q tests/test_flashfftconv.py

This test should run on machines with 40GB of GPU memory.

Short Depthwise Kernel

The short depthwise kernel is also installed with these commands. You can run this like this:

pytest -s -q tests/test_conv1d.py

How to Use FlashFFTConv

The flashfftconv package contains a PyTorch interface called FlashFFTConv, which is initialized with a particular FFT size.

This module computes an FFT convolution (iFFT(FFT(u) * FFT(k))).

python
from flashfftconv import FlashFFTConv

Usage:

python
from flashfftconv import FlashFFTConv

# size of the FFT
my_flashfftconv = FlashFFTConv(32768, dtype=torch.bfloat16) # generally more stable!

# B is batch size, H is model dimension, L is sequence length
B = 16
H = 768
# input can be smaller than FFT size, but needs to be divisible by 2
L = 16384

# the input, B H L
x = torch.randn(B, H, L, dtype=torch.bfloat16) # same type as the input
k = torch.randn(H, L, dtype=torch.float32) # kernel needs to be fp32 for now

out = my_flashfftconv(x, k)

Example Model

We recommend creating one FlashFFTConv object per model, and reusing it between layers.

For example:

…

Gating and Implicit Padding

A common use case for long FFT convolutions is for language modeling. These architectures often use gated convolutions and pad the inputs with zeros to ensure causality.

For example, a gated causal convolution might look like this in PyTorch:

…

Each of these padding operations, as well as the gating operations, incur expensive memory I/Os, which slows down the model.

FlashFFTConv supports implicit padding and gating without the need for extra I/O:

python
L = ... # get L from somewhere
flashfftconv = FlashFFTConv(2 * L, dtype=torch.bfloat16) # bf16 is usually necessary for gating
y = flashfftconv(u, k, in_gate, out_gate)

Short Depthwise Convolutions

For short, depthwise convolutions (groups = dimension in PyTorch Conv1D), you can run them like this:

…

To support mixed precision training, FlashDepthWiseConv1d supports using fp32 weights with fp16 inputs (or fp32 inputs). Currently the bf16 backward pass has a bug, but the forward pass is supported.

Benchmarking

FlashFFTConv Benchmarks

To run FlashFFTConv benchmarks, install the module and run python benchmarks/benchmark_flashfftconv.py.

These are the runtimes we see for a gated convolution for various sequence lengths, on one H100-SXM. All results scaled to batch size 64, hidden dimension 768.

Sequence Length 256 1K 4K 8K 16K 32K 1M 2M 4M
PyTorch 0.62 2.30 9.49 19.4 29.9 84.8 3,071.4 6,342.6 13,031.2
FlashFFTConv 0.11 0.29 1.43 3.58 12.2 26.3 1,768.9 4,623.5 10,049.4
Speedup 5.64× 7.93× 6.64× 5.42× 2.45× 3.22× 1.74× 1.37× 1.30×
Memory Savings 6.65× 6.40× 6.35× 6.34× 6.17× 5.87× 2.82× 2.81× 2.81×

Please see our paper for more benchmarks!

Short Depthwise Convolution Benchmarks

To benchmark short depthwise convolutions, install the module and run python benchmarks/benchmark_conv1d.py.

Here are some results for BLH input on H100:

B L D K torch time (ms) cudatime (ms) speedup
16 1024 768 5 0.19 0.03 5.50
16 1024 1024 5 0.25 0.04 6.00
16 1024 2048 5 0.50 0.08 6.50
16 1024 8192 5 2.08 0.29 7.21
16 2048 768 5 0.37 0.06 5.91
16 2048 1024 5 0.50 0.08 6.33
16 2048 2048 5 1.00 0.15 6.77
16 2048 8192 5 4.17 0.57 7.31
16 4096 768 5 0.74 0.12 6.17
16 4096 1024 5 0.99 0.15 6.56
16 4096 2048 5 2.03 0.29 7.04
16 4096 8192 5 8.25 1.14 7.27
16 8192 768 5 1.49 0.23 6.36
16 8192 1024 5 2.01 0.30 6.80
16 8192 2048 5 4.10 0.57 7.18
16 8192 8192 5 16.42 2.26 7.26

We also support BHL input, but it's a bit slower (still optimizing!).

Input Requirements and Notes

Currently, we have a few requirements on the inputs to the interface to FlashFFTConv:

  • We assume that the input u has shape (B, H, L), and the kernel k has shape (H, L).
  • These inputs must be contiguous in GPU memory (u.is_contiguous() should be True).
  • The FFT size (seqlen that FlashFFTConv is initialized with) must be a power of two between 256 and 4,194,304.
  • For FFT sizes larger than 32,768, H must be a multiple of 16.
  • L can be smaller than FFT size but must be divisible by 2. For FFT sizes 512 and 2048, L must be divisible by 4.
  • We only support FP16 and BF16 for now. FP16 is faster, but we generally find BF16 more stable during training.
  • For short depthwise convs, we only support FP16 for now, and the kernel size has to be odd.

Citation

This work builds on a line of work studying how to make FFT convolutions efficient on GPUs:

  • Monarch Mixer: A Simple Sub-Quadratic GEMM-Based Architecture
  • Simple Hardware-Efficient Long Convolutions for Sequence Modeling
  • Hungry Hungry Hippos: Towards Language Modeling with State Space Models

If you use this codebase or otherwise found the ideas useful, please reach out to let us know - we love hearing about how our work is being used! You can reach Dan at [email protected].

You can also cite our work:

…

Issues· 0 开放

查看全部 Issues在 GitHub 打开

暂无开放 Issues,或尚未同步最近议题。

> 标签

C++

暂无评论,来聊聊你的看法吧

> 工具信息

发布日期2026年8月1日
最后更新2026年9月17日
分类编程语言
定价开源

> 相关工具

T
TypeScript
JavaScript 的超集,为前端与全栈提供静态类型
P
Python
通用编程语言,广泛用于 Web、数据与 AI
G
Go
Google 推出的简洁高效系统语言