#247·xsv

Transparent decompression

Author: rigtorpCreated Oct 27, 2020Updated Feb 10, 2025

First I want to thank you for creating this great tool. It has helped me work with bigger than main memory datasets that Pandas could not handle.

I like to store my big csv files compressed using zstd. A great feature in xsv would be to transparently decompress these files when read. I propose to detect if the input is a gzip or zstd file using the file magic. We can of course use a shell pipeline to decompress files on the fly, but that will only work for commands that only have one input.

Is this a reasonable thing to add? It's something I might work on at some point in time.