BitNet

开源

1- 位 LLMs 的正式推断框架

39.8K stars0 点赞0 次浏览

工具介绍

1- 位 LLMs 的正式推断框架

Overview bitnet.cpp是1-bit LLMs(如BitNet b1.58)的官方推断框架. 它提供一组优化的内核,支持快速而无损失地推断出CPU和GPU上的1.58位模型(NPU支持即将到来). 尝试通过这个在线演示,或者在自己的CPU或GPU上构建和运行. bitnet.cpp在ARM CPU上实现了1.37x到5.07x的速度,较大的模型的性能增益更大. 此外,它将能耗减少55.4%至70.0%,进一步提高了总体效率。在x86CPU上,速度从2.37x到6.17x不等,能量削减从71.9%到82.2%. 此外,bitnet.cpp可以在单个CPU上运行一个100B BitNet b1.58模型,实现与人读(每秒5-7个令牌)相仿的速度,大大提高了在本地设备上运行LLM的潜力. 详见技术报告。玛..

核心特点

•Fast CPU Inference: Achieves up to 6.17x speedup on x86 CPUs and 5.07x on ARM CPUs compared to full-precision models.
•Energy Efficient: Reduces energy consumption by up to 82.2% on x86 and 70.0% on ARM.
•GPU Support: Official GPU inference kernel available for accelerated deployment.
•Chat-Ready: Supports conversational mode for interactive use.
•1.42x to 2.28x speedup over F16 on prefill (8 threads, x86)
•Lossless Quality: Competitive embedding quality with 2 bits per weight
•I2_S Kernel: Supports optimized I2_S conversion on x86 CPUs
•1.32x to 1.74x speedup over F16 on prefill (8 threads, x86)
•Lossless Quality: Competitive embedding quality with 2 bits per weight
•Lightweight: Only 270M parameters for edge deployment scenarios