Feature request: native multithreaded decompression for standard frames
Is your feature request related to a problem? Please describe.
Some workloads compress a large archive once and decompress it many times. Zstandard supports multithreaded compression through -T#, but decompression remains single-threaded and can become a bottleneck on modern multicore systems with fast storage.
In one test with an approximately 4 GB archive, multithreaded zstd compression took about 21 seconds, while single-threaded zstd decompression plus tar extraction took about 26 seconds, compared with about 31 seconds using pigz. Compression completing faster than the extraction path is a useful signal that decompression is worth optimizing, although the extraction measurement also includes tar and filesystem writes.
This is a follow-up to #2470, where several users requested that multithreaded decompression be reconsidered after the issue was closed. Related discussions also exist in #2245 and #2499.
Describe the solution you would like
Add opt-in multithreaded decompression to the standard zstd CLI and API, ideally for streams produced by the normal zstd encoder. A CLI interface such as zstd -d -T0 would be intuitive and consistent with compression.
Describe alternatives you have considered
pzstd, multiple independent frames, or manually splitting the input can enable parallel decompression, but these approaches require a different encoding strategy and may affect compression ratio or integration complexity.
Source: facebook/zstd