Question about ROLLING KV CACHE WITH ATTENTION SINKS

Hello,

Thank you very much for the excellent work and code. I had a question about the rolling KV cache. Does the cache remain valid beyond layer one when decoding in autoregressive fashion? I understand that the positional embeddings are applied at every step to ensure the model always sees consistent positions but when the window is shifted the KV cache in the second layer was computed based on the output of the first layer which still contains dependencies on the previous step attention which in turn relied on the previous step's position embeddings.

Is the KV caching supposed to be approximate and not exact? I would really appreciate any guidance or clarification.

Thank you, Deep