Feature Request: Support LFS upload in chunks
Describe the issue
I use Git LFS to manage a large project that primarily stores continuously-generated CSV data and log files. My workflow requires adding, committing, and pushing these files to a Git LFS server, then pulling them on other servers for analysis — all while preserving full traceability of every file's change history.
Because the CSV and log data is generated continuously, each commit involves increasingly large files. Even when only a small portion of data is newly appended, the entire file must be re-uploaded to the LFS server, since Git LFS treats each file as an opaque blob identified by its SHA-256 hash. This results in significant redundant bandwidth and storage overhead.
Describe the solution you'd like
I would like Git LFS to support uploading files in chunks (e.g., 4 MiB fixed-size blocks). When a new commit is made, only the new or modified chunks would need to be uploaded, rather than the entire file. This could work similarly to how deduplication or block-based transfer protocols operate:
- A large file is split into fixed-size chunks on the client side.
- Each chunk is hashed and uploaded independently.
- The LFS server reconstructs the full file from its constituent chunks.
- On subsequent commits where data is appended, only the new chunks are transferred.
Describe alternatives you've considered
- Manual file splitting: Splitting large CSV/log files manually before committing. However, this breaks the logical integrity of the data and complicates downstream analysis.
- Incremental file rotation: Rotating files at fixed intervals (e.g., hourly logs). This reduces per-file size but doesn't eliminate the problem of re-uploading large rotated files, and it shifts the burden to the application layer.
- External deduplication tools: Using tools like
resticorborgfor deduplicated backups. However, these don't integrate with Git's version control workflow, losing the traceability that Git LFS provides.
Additional context
This feature would be particularly valuable for use cases involving continuously-growing append-only files, such as:
- Time-series CSV datasets
- Application/server log files
- Machine learning training logs
- IoT sensor data exports
Block-level deduplication would dramatically reduce upload time, bandwidth consumption, and server-side storage costs for these scenarios, while maintaining Git's versioning and traceability benefits.
Source: git-lfs/git-lfs