Sortformer splits one speaker into two when their distance to the mic changes

Is your feature request related to a problem? Please describe.

Hello, I've been running diar_streaming_sortformer_4spk-v2 on recordings where one person moves around the room while talking.

Whenever the speaker gets noticeably farther from or closer to the mic, the model tends to open a new speaker slot for them, so one person ends up as speaker_0 for the first half and speaker_2 for the rest.

Describe the solution you'd like

I'm planning to fine-tune from v2 on LibriSpeech mixtures (1-4 speakers, 90s) where half the sessions are dry and half are RIR-convolved with the speaker position changing mid-session, plus MUSAN noise. Basically the RIR augmentation idea from VibeVoice-ASR-streaming applied to diarization.

AMI SDM DER as a regression check, plus a simple "how many labels does one reference speaker get" count on the distance sets.

Before I sink GPU time into this I'd like to know whether that direction makes sense to you, or whether there's a better way to handle it.

Describe alternatives you've considered

Additional context