why recompute can differ from window attention?
Author: habaohabaCreated Oct 11, 2024Updated Nov 11, 2024
I think recompute just give the same value of kv state which is saved when using window attention. So what is the difference between recompute and cache version of slide window? Or it is because no matter what position embedding we use, llm juse learn to set first index a large attention value?
Source: mit-han-lab/streaming-llm