Filter, Sort & Delete Duplicate Files Recursively
Filter, Sort & Delete Duplicate Files Recursively
Deduplicator
Find, Sort, Filter & Delete duplicate files
[!NOTE] This project is maintained with the assistance of AI tools. All changes are subject to manual review and a comprehensive test suite to ensure stability and quality.
……Currently, you can only install deduplicator using cargo package manager.
GxHash relies on aes hardware acceleration, so please set
RUSTFLAGSto"-C target-feature=+aes"or"-C target-cpu=native"before installing.
$ RUSTFLAGS="-C target-cpu=native" cargo install deduplicator
# or
$ RUSTFLAGS="-C target-feature=+aes,+sse2" cargo install deduplicator$ RUSTFLAGS="-C target-cpu=native" cargo install deduplicator --git https://github.com/sreedevk/deduplicator
# or
$ RUSTFLAGS="-C target-feature=+aes,+sse2" cargo install --git https://github.com/sreedevk/deduplicatortar -zxvf .tar.gz $PATH. /usr/local/bin/.Deduplicator uses size comparison and GxHash to quickly check a large number of files to find duplicates. its also heavily parallelized. The default behavior of deduplicator is to only hash the first page (4K) of the file. This is to ensure that performance is the default priority. You can modify this behavior by using the --strict flag which will hash the whole file and ensure that 2 files are indeed duplicates. I'll add benchmarks in future versions.
I've used hyperfine to run deduplicator on files generated by the rake file at rakelib/benchmark.rake. The Benchmarking accuracy can further be improved by isolating runs inside restricted docker containers. I'll include that in the future. For now, here's the hyperfine output on my i7-12800H laptop with 32G of RAM.
……parallelization
user supplied cache file path for faster re-runs
hardlinks / symlinks support
max file path size should use the last set of duplicates
add more unit tests
restore json output (was removed in 0.3 due to quality issues)
fix memory leak on very large filesystems
tui
change the default hashing method to include the first & last page of a file (8K)
provide option to localize duplicate detection to arbitrary levels relative to current directory
localize file meta store locks to sub path levels to avoid global lock contention from multiple threads.
bulk operations
fix: partial hash collision - a file full of null bytes ("\0") and an empty file. This is a known trade off in gxhash.
potential optimizations
No open issues yet, or sync has not completed.