用于通过位元压缩整数压缩的 SIMD 算法。此 crate 是 simdcomp C 库的移植版本。
This crate is a Rust port of Daniel Lemire's simdcomp C library.
It makes it possible to compress/decompress :
:star: It is fast. Expect > 4 billions integers per seconds.
bitpacking compiles on stable rust but require rust > 1.27 to compile.
Just add to your Cargo.toml :
bitpacking = "0.9"For some bitpacking flavor and for some platform, the bitpacking crate may benefit from some specific simd instruction set.
In this case, it will always ship an alternative scalar implementation and will fall back to the scalar implementation at runtime.
In other words, your do not need to configure anything. Your program will run correctly, and at the fastest speed available for your CPU.
Traditional compression schemes like LZ4 are not really suited to address this problem efficiently. Instead, there are different families of solutions to this problem.
One of the most straightforward and efficient ones is bitpacking :
128 when using the SSE2 implementation).b that makes it possible to represent all the integers.
In other words, the smallest b such that all integers in the block are stricly smaller than 2b.b-bits.For instance, assuming a block of 4, when encoding 4, 9, 3, 2. Assuming that the highest value in the block is 9, b = 4. All values will then be encoded over 4 bits as follows.
| original number | binary representation |
|---|---|
| 4 | 0100 |
| 9 | 1001 |
| 3 | 0011 |
| 2 | 0010 |
| ... | ... |
As a result, each integer of this block will only require 4 bits.
:warning: BitPacker1x, BitPacker4x, and BitPacker8x produce different formats,
and are incompatible one with another.
BitPacker4x and BitPacker8x are designed specifically to leverage SSE3 and AVX2
instructions respectively.
It will safely fall back at runtime to a scalar implementation of these format if these instruction sets are not available on the running CPU.
:ok_hand: I recommend using BitPacker4x if you are in doubt.
BitPacker1x is what you would expect from a bitpacker.
The integer representation over b bits are simply concatenated one
after the other. One block must contain 32 integers.
BitPacker4x bits ordering works in layers of 4 integers. This gives an opportunity
to leverage SSE3 instructions to encode and decode the stream.
One block must contain 128 integers.
BitPacker8x bits ordering works in layers of 8 integers. This gives an opportunity
to leverage AVX2 instructions to encode and decode the stream.
One block must contain 256 integers.
…The following benchmarks have been run on one thread on my laptop's CPU: Intel(R) Core(TM) i5-8250U CPU @ 1.60GHz.
You can get accurate figures on your hardware by running the following command.
cargo bench| operation | throughput |
|---|---|
| compress | 1.4 billions int/s |
| compress_delta | 1.0 billions int/s |
| decompress | 1.8 billions int/s |
| decompress_delta | 1.4 billions int/s |
| operation | throughput |
|---|---|
| compress | 5.3 billions int/s |
| compress_delta | 2.8 billions int/s |
| decompress | 5.5 billions int/s |
| decompress_delta | 5 billions int/s |
| operation | throughput |
|---|---|
| compress | 7 billions int/s |
| compress_delta | 600 millions int/s |
| decompress | 6.5 billions int/s |
| decompress_delta | 5.6 billions int/s |
暂无开放 Issues,或尚未同步最近议题。