BitBLAS

BitBLAS

BitBLAS is a library to support mixed-precision BLAS operations on GPUs, for example, the $W_{wdtype}A_{adtype}$ mixed-precision matrix multiplication where $C_{cdtype}[M, N] = A_{adtype}[M, K] \times W_{wdtype}[N, K]$. BitBLAS aims to support efficient mixed-precision DNN model deployment, especially the $W_{wdtype}A_{adtype}$ quantization in large language models (LLMs), for example, the $W_{UINT4}A_{FP16}$ in GPTQ, the $W_{INT2}A_{FP16}$ in BitDistiller, the $W_{INT2}A_{INT8}$ in BitNet-b1.58. BitBLAS is based on techniques from our paper "Ladder: Enabling Efficient Low-Precision Deep Learning Computing through Hardware-aware Tensor Transformation" at OSDI'24.

Some of the key features of BitBLAS include:

• High performance matrix multiplication for both GEMV (e.g., the single batch auto-regressive decode phase in LLM) and GEMM (e.g., the batched auto-regressive decode phase and the prefill phase in LLM):
  • $W_{wdtype}A_{adtype}$ mixed-precision matrix multiplication including FP16xFP8/FP4/INT4/2/1, INT8xINT4/2/1, etc. Please checkout support matrix for detailed data types support.
  • Matrix multiplication like FP16xFP16 and INT8xINT8.
• Auto-Tensorization for TensorCore-like hardware instructions.
• Implemented integration to PyTorch, GPTQModel, AutoGPTQ, vLLM and BitNet-b1.58 for LLM deployment. Please checkout benchmark summary for detailed end2end LLM inference performance.
• BitBLAS first implemented $W_{INT2}A_{INT8}$ GEMV/GEMM in BitNet-b1.58 with 8x/2x speedup over cuBLAS $W_{FP16}A_{FP16}$ on A100, please checkout op_benchmark_a100_int2_scaling for detailed benchmark results. Please checkout BitNet-b1.58 integration for the integration with the 3rdparty reproduced BitNet-b1.58 model.
• Support customizing mixed-precision DNN operations for your specific scenarios via the flexible DSL (TIR Script).

Latest News

• 11/04/2024 : We've supported high performance A INT4 x W INT4/INT2 Matmul for BitNet a4.8.
• 10/02/2024 : We've added initial Flash Attention Ops and its implementation in Tilelang! Please refer to PythonAPI and QuickStart docs and PR #202.
• 08/12/2024 : We've improved performance for contiguous batching. To enable it, you'll need to set specific flags. For more details, please refer to PR #133.
• 07/11/2024 ✨: Ladder is published and presented in OSDI'24. Please find Ladder paper and presentation if you are interested in the technical details of BitBLAS.
• 06/25/2024 : BitBLAS has been integrated into GPTQModel! You can now use BitBLAS as a backend in GPTQ.
• 05/04/2024 : We’ve added integration examples for the 1.58-bit model! Check out the files under integration/BitNet.
• 04/30/2024 : BitBLAS now supports FP8 TensorCore ($W_{E5M2/E4M3}A_{E4M3/E5M2}$), providing more combinations beyond the three available in cuBLAS!
• 04/19/2024 ✨: We are excited to announce that BitBLAS, a high-performance library for mixed-precision DNN model deployment, is now open source and available to the public!

Integration Example of FasterTransformer with BitBLAS

Benchmark Summary

BitBLAS achieves exceptional performance across a variety of computational patterns. Below are selected results showcasing its capabilities:

• End2End Integration with Quantize Inference Kernel for AutoGPTQ and vLLM.

• Weight Only Matmul performance on A100

• TensorCore FP16/INT8 GEMM Performance Vs. Vendor Library on A100 and RTX4090

For more detailed information on benchmark sets with other formats (NF4/FP4) and other devices (RTX 3090), please refer to the benchmark.

Support Matrix

We are continuously expanding the support matrix. If you have any specific requirements, please feel free to open an issue or PR.

Getting Started with an Example

Installing with pip

Prerequisites for installation via wheel or PyPI

• Operating System: Ubuntu 20.04 or later
• Python Version: >= 3.8
• CUDA Version: >= 11.0

The easiest way to install BitBLAS is directly from the PyPi using pip. To install the latest version, run the following command in your terminal.

pip install bitblas

Alternatively, to install the latest version of BitBLAS from the github repository, you can run the following command:

pip install git+https://github.com/microsoft/BitBLAS.git

After installing BitBLAS, you can verify the installation by running:

python -c "import bitblas; print(bitblas.__version__)"  

Note: Currently, BitBLAS whl is only supported on Ubuntu 20.04 or later version as we build the whl files on this platform. Currently we only provide whl files for CUDA>=11.0 and with Python>=3.8. If you are using a different platform or environment, you may need to build BitBLAS from source. More installation methods can be found in the installation document.

Example: $W_{INT4}A_{FP16}$ mixed-precision matrix multiplication

BitBLAS provides two Python APIs to perform mixed-precision matrix multiplication:

• bitblas.Matmul implements the $W_{wdtype}A_{adtype}$ mixed-precision matrix multiplication of $C_{cdtype}[M, N] = A_{adtype}[M, K] \times W_{wdtype}[N, K]$ where $W_{wdtype}$ indicates the weight of $wtype$, A_{adtype} indicates the activation of $adtype$, and C_{cdtype} indicates the output of $cdtype$.
• bitblas.Linear is a PyTorch nn.Linear-like module to support a Linear of mixed-precision.

Here is an example for a $W_{INT4}A_{FP16}$ mixed-precision matrix multiplication: $out_{FP16}[M, N] = A_{FP16}[M, K] \times W_{INT4}[N, K]$, this example includes the creation of input matrices, quantization of weight matrices, and execution of the matrix multiplication with the bitblas.Matmul API. The result is then compared against a reference result obtained through conventional methods to ensure accuracy.

…

Note: More examples can be found in the QuickStart document.

Documents

• Installation: The installation document of BitBLAS. Make sure you already have the cuda toolkit (version >= 11.0) installed in the system.

  • You can easily install from pip install bitblas from PyPi. Currently we only provide whl files for CUDA>=11.0 and Ubuntu>=20.04 with Python>=3.8, if you are using a different version of CUDA or OS environment, you may need to build BitBLAS from source.

• QuickStart: This document provides examples to use BitBLAS in your program with bitblas.Matmul and bitblas.Linear.

• Python API: The Python API document of BitBLAS. BitBLAS provides two Python APIs to perform mixed-precision matrix multiplication:

  • bitblas.Matmul i