Why not interleave sin/cos as in the original Transformer positional encoding?

https://github.com/lucidrains/denoising-diffusion-pytorch/blob/1d9d8dffb72e02172da8a77bee039b1c72b7c6d5/denoising_diffusion_pytorch/denoising_diffusion_pytorch.py#L126

Hi, thanks for the great work!

I was reading the SinusoidalPosEmb implementation, and I noticed that the sinusoidal encoding is constructed by concatenating sin and cos along the last dimension:

emb = torch.cat((emb.sin(), emb.cos()), dim=-1)

However, in the original Transformer paper, the positional encoding is defined in an interleaved fashion: pe = torch.zeros(t.shape[0], dim, device=t.device) pe[:, 0::2] = emb.sin() pe[:, 1::2] = emb.cos()

This produces [sin, cos, sin, cos, ...], whereas the current implementation results in [sin, sin, ..., cos, cos, ...].

Is there a specific reason for this deviation from the original formula? Does the current implementation have empirical or theoretical advantages, or is it just a simplified design for downstream processing?

Thanks in advance!