Silently slower delta-rule fallback when training a Qwen3.8-27B LoRA with examples/sft under FSDP on 8x B200

System Info

peft 0.20.1.dev0 (main at f5ea6b00), transformers 5.16.1, accelerate 1.14.0, trl 1.12.0, torch 2.14.0+cu130, Python 3.11.16, Linux, one node with 8x NVIDIA B200, CUDA 13.0. No flash-attn, fla or hub kernels installed.

Who can help?

No response

Reproduction

I ran examples/sft/train.py as in examples/sft/run_peft_fsdp.sh (8-process FSDP, bf16) on Qwen/Qwen3.8-27B with LoRA r=8 and gradient accumulation 4, on 8x B200, with peft installed from main and no fla or flash-attn, and got 52.8 s per optimizer step. A profile showed the 48 linear-attention layers running transformers' pure-torch delta-rule fallback (transformers/models/qwen3_5/modeling_qwen3_5.py:294-316), which writes each chunk's result into a slice of the output, so the backward was mostly copy and select nodes. With the fallback reading chunk views instead of writing into slices, the same step took 41.7 s (from 53.0 s in that measurement run); a smaller second change, SHARD_GRAD_OP in the FSDP config, took it to 35.8 s.

I opened #3712 to record my experiment: setup, measurements, traces and some potential fixes. A different fix may also well suit the codebase.

Expected behavior

Nothing in the logs said which delta-rule implementation was in use, so I only found it by profiling. One log line at model load naming the delta-rule path that was picked would have saved me the profile.