Please do not abandon this project!
Author: oobaboogaCreated Nov 27, 2023Updated May 12, 2024
Earlier this year I was impressed with the offloading performance of FlexGen, and I wonder how it would compare with the performance currently provided by llama.cpp for Llama and Llama-2 models in a CPU offloading scenario.
Any chance Llama support could be added to FlexGen @Ying1123 @keroro824?
Source: FMInference/FlexLLMGen