dwarfs

DwarFS

The Deduplicating Warp-speed Advanced Read-only File System.

A fast high-compression read-only file system for Linux, FreeBSD, macOS and Windows.

Table of contents

• What is DwarFS (in plain words)?
• Why not just use .zip or .tar.gz?
• Performance comparison overview
• Quick Start
• Overview
• History
• Building and Installing
  • Note to Package Maintainers
  • Prebuilt Binaries
  • Universal Binaries
  • Dependencies
  • Building
  • Installing
  • Static Builds
• Usage
• Using the Libraries
• Windows Support
  • Building on Windows
• macOS Support
• Use Cases
  • Astrophotography
• Dealing with Bit Rot
• Extended Attributes
• Comparison
  • With SquashFS
  • With SquashFS & xz
  • With lrzip
  • With zpaq
  • With zpaqfranz
  • With wimlib
  • With Cromfs
  • With EROFS
  • With fuse-archive
• Performance Monitoring
• Other Obscure Features
• Related projects
• Notable users
• Stargazers over Time

What is DwarFS (in plain words)?

DwarFS is a mountable archive: you pack a directory into one file (an "image"), then open it instantly like a normal folder — no full extraction, no temporary files. It’s read-only when mounted, so you can browse, open, and run files safely in place. This works particularly well for big collections with lots of similar or repeated content — think many versions of a project or dataset, folders of documents and text files, backups/snapshots that mostly overlap, or libraries with many near-duplicates. The image can be mounted in fractions of a second, you can use the contents immediately with no delay, and you extract only if you actually need to.

Why not just use .zip or .tar.gz?

Traditional archives are fine for storage, but they’re usually slow to open and awkward for random access (jumping around inside large files or across many files). DwarFS is built for fast random reads and space savings. It groups similar files and removes duplication, so images are often smaller than a simple tar/zip, while reads stay snappy — even when accessing lots of files simultaneously. In practice, you keep huge directories compressed, mount them in milliseconds, and work as if they were already unpacked.

Performance comparison overview

This comparison uses DwarFS 0.14.1, 7-Zip 25.00 (x64), and tar + pigz for all .tar.gz tests since plain gzip would have been incredibly slow. Both .tar.gz and 7zip archives were mounted using fuse-archive v1.16 with the -olazycache option since the default of caching the entire archive in memory is infeasible for large archives.

1139 complete Perl installations

[^pl1]: using pigz -9 [^pl2]: using zstd --long=31 --ultra -22 -T0 [^pl3]: not supported by fuse-archive [^pl4]: $ ls -1 mnt/*/perl*/bin/perl5* | xargs -d $'\n' -n1 -P16 sha256sum [^pl5]: killed after making no progress for 15 minutes [^pl6]: killed when only 78 files were finished after about 20 minutes [^pl7]: using -comp zstd -Xcompression-level 22 -b 1M -tailends; using squashfuse_ll 0.6.0 [^pl8]: $ fd -t f . mnt | wc -l

All artifacts from 205 DwarFS CI builds

[^ci1]: $ ls -1 mnt/*/dwarfs-*-Linux-x86_64-*-minsize*/bin/mkdwarfs | xargs -d $'\n' -n1 -P16 sh -c '$0 -h' [^ci2]: killed after making no progress for 15 minutes [^ci3]: killed after making no progress and consuming more than 32 GiB of RAM

Game audio assets from sonniss.com

[^wav2]: $ find mnt -type f | xargs -d $'\n' -n1 -P16 sha256sum [^wav3]: using -comp zstd -Xcompression-level 22 -b 1M -tailends

Quick Start

To create a DwarFS image from a directory, use mkdwarfs:

$ mkdwarfs -i /path/to/input/dir -o /path/to/image.dwarfs

To mount the image, use the dwarfs FUSE driver:

$ mkdir /path/to/mountpoint
$ dwarfs /path/to/image.dwarfs /path/to/mountpoint
$ ls /path/to/mountpoint

To extract the image, use dwarfsextract:

$ dwarfsextract -i /path/to/image.dwarfs -o /path/to/output/dir

There's also the dwarfsck tool that can be used for a variety of tasks, for example listing the files in a DwarFS image:

$ dwarfsck /path/to/image.dwarfs -l

It can also be used for generating checksums for each file in the image in a format that is recognized by the *sum commands:

$ dwarfsck /path/to/image.dwarfs --checksum=sha512 | sha512sum --check

This is useful if you want to verify the integrity of the actual files stored in the DwarFS image.

Overview

DwarFS is a read-only file system with a focus on achieving very high compression ratios, particularly for highly redundant data.

This probably doesn't sound very exciting, because if it's redundant, it should compress well. However, I found that other read-only, compressed file systems don't do a very good job at making use of this redundancy. See here for a comparison with other compressed file systems.

DwarFS also doesn't compromise on speed; in my use cases, it performs on par with, or better than, SquashFS. For my primary use case, DwarFS compression is an order of magnitude better than SquashFS compression, it's 6 times faster to build the file system, it's typically faster to access files on DwarFS and it uses less CPU resources.

To give you an idea of what DwarFS is capable of, here's a quick comparison of DwarFS and SquashFS on a set of video files with a total size of 39 GiB. The twist is that each unique video file has two sibling files with a different set of audio streams (this is an actual use case). So there's redundancy in both the video and audio data, but as the streams are interleaved and identical blocks are typically very far apart, it's challenging to make use of that redundancy for compression. SquashFS essentially fails to compress the source data at all, whereas DwarFS is able to reduce the size to nearly one-third, which is close to the theoretical maximum:

$ du -hs dwarfs-video-test
39G     dwarfs-video-test
$ ls -lh dwarfs-video-test.*fs
-rw-r--r-- 1 mhx users 14G Jul  2 13:01 dwarfs-video-test.dwarfs
-rw-r--r-- 1 mhx users 39G Jul 12 09:41 dwarfs-video-test.squashfs

Furthermore, when mounting the SquashFS image and performing a random-read throughput test using fio-3.34, both squashfuse and squashfuse_ll top out at around 230 MiB/s:

$ fio --readonly --rw=randread --name=randread --bs=64k --direct=1 \
      --opendir=mnt --numjobs=4 --ioengine=libaio --iodepth=32 \
      --group_reporting --runtime=60 --time_based
[...]
   READ: bw=230MiB/s (241MB/s), 230MiB/s-230MiB/s (241MB/s-241MB/s), io=13.5GiB (14.5GB), run=60004-60004msec

In comparison, DwarFS manages to sustain random read rates of 20 GiB/s:

  READ: bw=20.2GiB/s (21.7GB/s), 20.2GiB/s-20.2GiB/s (21.7GB/s-21.7GB/s), io=1212GiB (1301GB), run=60001-60001msec

Distinct features of DwarFS are:

• Clustering of files by similarity using a similarity hash function. This makes it easier to exploit the redundancy across file bounda