Activations in AdaLN
Author: Piyush-555Created Apr 1, 2025Updated Apr 1, 2025
I wonder if there's a specific reason for having an activation function before the linear projection in AdaLN. Would it be better to have particular activations after the linear projection in addition to that? E.g., sigmoid for the gate, positive for scale, identity for shift.
Source: facebookresearch/DiT