How to split execution of prefill and decode for Flexgen?

Author: sunchaeskCreated Jul 9, 2024Updated Jul 9, 2024

I was wondering what would be the best way to split the execution of flexgen to prefill and decode only.

How should I save the values from prefill and how should I load them when I am running flexgen again for decode only.

Thanks