why `max_gen_len` is needed when considering `space_needed`?
Author: Mr-lonely0Created Mar 29, 2024Updated Jan 4, 2025
Thank you for your time, here are my questions:
- I wonder why cache_size is updated only at every turn rather than every decoding step.
- When considering space needed in cache, why max_gen_len is needed to include
Source: mit-han-lab/streaming-llm