why `max_gen_len` is needed when considering `space_needed`?

Author: Mr-lonely0Created Mar 29, 2024Updated Jan 4, 2025

Thank you for your time, here are my questions:

  1. I wonder why cache_size is updated only at every turn rather than every decoding step.
  2. When considering space needed in cache, why max_gen_len is needed to include

Source: mit-han-lab/streaming-llm