Severe Training Slowdown – xFormers Ignored, SDPA Used Instead

Hi there,

I’ve run into a major performance issue after switching to PyTorch ≥ 2.2 while training SDXL LoRA models with Kohya (using Prodigy optimizer and 2048px images).

Before: ~4–5h training time for 1800 steps (60 images, batch 1) Now: 17h+, despite xFormers being selected in the GUI.

Observations: Console shows:

torch.autograd.graph.py:825: UserWarning: cuDNN SDPA backward got grad_output.strides() != output.strides() ...

Training clearly uses cuDNN SDPA, not xFormers.

SDPA is causing massive overhead on my system (Windows 11, RTX 4060 Ti, 16GB VRAM).

❗ Expected Behavior: If xFormers is selected in the GUI, it should be strictly prioritized. Fallbacks to SDPA or other attention backends should be opt-in only, or at least clearly communicated in logs/UI.

✅ Request: Please add a force-disable for SDPA and/or a clear toggle between attention backends.

Clarify if PyTorch >=2.2 auto-enables SDPA even when xFormers is active, and how to prevent that.

This change would help avoid 10+ hour training delays and ensure users get the expected performance out of their hardware.

Thanks for your awesome work – just want to help others avoid this trap!