Can we benefit from using streaming attention computation during pretraining?

Hi, in your paper you discuss adding a learnable placeholder token (sink token) and adopting full attention computation. I would like to ask whether it is possible—and potentially beneficial—to use streaming attention computation during pretraining.