缓存损失 (GradCache) 会触发每个列每个小批量的一个 DDP 全部归约,而不是每个步骤一个

作者: BramVanroy创建于 2026年9月15日更新于 2026年9月17日
标签bug

`surrogate.backward()` runs once per mini-batch, AND once per sentence-feature column (anchor, positive, ...). The problem is that model.no_sync() is never called. When the `loss_obj.model` is wrapped in DistributedDataParallel, every one of these calls is a forward+backward through the DDP module, so DDP's bucket-ready hooks fire and dispatch a real gradient all-reduce on every single mini-batch and not just once per optimizer step, which is what we would expect to happen.

内容来源: huggingface/sentence-transformers