make_pe() in class RelPosEncXL
Author: coollipCreated Jul 27, 2026Updated Aug 9, 2026
Labelsbug
Describe the bug
I noticed that the relative sinusoidal positional encoding in make_pe() is defined as (https://github.com/speechbrain/speechbrain/blob/develop/speechbrain/nnet/attention.py#L397):
sinusoids = torch.sin(positions * self.inv_freq)
pe_past[:, 0::2] = sinusoids
pe_past[:, 1::2] = torch.cos(positions * self.inv_freq)
pe_future[:, 0::2] = sinusoids # same for past and future
pe_future[:, 1::2] = torch.cos(-positions * self.inv_freq)Why are sinusoids the same for past and future when sin() is odd and cos() is even?
I also noticed that in espnet it is defined as (https://github.com/espnet/espnet/blob/master/espnet2/legacy/nets/pytorch_backend/transformer/embedding.py#L305):
pe_positive[:, 0::2] = torch.sin(position * div_term)
pe_positive[:, 1::2] = torch.cos(position * div_term)
pe_negative[:, 0::2] = torch.sin(-1 * position * div_term)
pe_negative[:, 1::2] = torch.cos(-1 * position * div_term)Expected behaviour
check the code
To Reproduce
No response
Environment Details
No response
Relevant Log Output
Additional Context
No response
Source: speechbrain/speechbrain