百科.dev
全部条目在线工具AI 编程趋势榜开源项目技术资讯提交条目
登录
< 返回工具列表
Z

zfsbackup-go

> 编程语言
开源

将 ZFS 快照备份到 Google、Amazon、Azure 等云存储服务中。以企业为设计理念。

377 stars0 点赞0 次浏览
访问官网GitHub

工具介绍

将 ZFS 快照备份到 Google、Amazon、Azure 等云存储服务中。以企业为设计理念。

ZFSBackup

DISCLAIMER: This is a work in progress in still considered beta though I personally use this in a production environment and have tested it for my own use cases (looking for feedback on other people's experience before considering this "production ready").

Overview

This backup software was designed for the secure, long-term storage of ZFS snapshots on remote storage. Backup jobs are resilient to network failures and can be stopped/resumed. It works by splitting the ZFS send stream (the format for which is committed and can be received on future versions of ZFS as per the man page) into chunks and then optionally compresses, encrypts, and signs each chunk before uploading it to your remote storage location(s) of choice. Backup chunks are validated using SHA256 and CRC32C checksums (along with the many integrity checks builtin to compression algorithms, SSL/TLS transportation protocols, and the ZFS stream format itself). The software is completely self-contained and has no external dependencies.

This project was inspired by the duplicity project.

Highlights

  • Written in Go
  • No external dependencies - Just drop in the binary on your system and you're all set!
  • Backup jobs are resumeable and resilient to network failures
  • Backup files can be compressed and optionally encrypyted and/or signed.
  • Concurrent by design, enable multiple cores for parallel processing
  • Configurable Operation - Limit bandwidth usage, space usage, CPU usage, etc.
  • Backup to multiple destinations at once, just comma separate destination URIs
  • Uses familiar ZFS send/receive options

Supported Backends

  • Google Cloud Storage (gs://)
    • Auth details:
    • 99.999999999% durability - Using erasure encodings
  • Amazon AWS S3 (s3://) (Glacier supported indirectly via lifecycle rules)
    • Auth details:
    • 99.999999999% durability - Using replication and checksums on the data for integrity validation and repair
  • Any S3 Compatible Storage Provider (e.g. Minio, StorageMadeEasy, Ceph, etc.)
    • Set the AWS_S3_CUSTOM_ENDPOINT environmental variable to the compatible target API URI
  • Azure Blob Storage (azure://)
    • Auth: Set the AZURE_ACCOUNT_NAME and AZURE_ACCOUNT_KEY environmental variables to the appropiate values or if using SAS set AZURE_SAS_URI to a container authorized SAS URI
    • Point to a custom endpoint by setting the AZURE_CUSTOM_ENDPOINT envrionmental variable
    • Although no durability target is provided, there is an in-depth explanation of their architecture here - Using the Reed-Solomon erasure encoding and user-configurable redundancy settings
  • BackBlaze B2 (b2://)
    • Auth: Set the B2_ACCOUNT_ID and B2_ACCOUNT_KEY environmental variables to the appropiate values
    • 99.999999999% durability - Using the Reed-Solomon erasure encoding
  • Local file path (file://[relative|/absolute]/local/path)
  • SSH/SFTP (ssh://)
    • Auth: username & password, public key or ssh-agent.
    • For username & password set the SSH_USERNAME and SSH_PASSWORD environment variables or use the url format: ssh://username:[email protected]/remote/path.
    • For public key auth set the SSH_KEY_FILE environment variable. By default zfsbackup tries to use common key names from the users home directory.
    • ssh-agent auth is activated when SSH_AUTH_SOCK exists.
    • By default zfsbackup also uses the known hosts file from the users home directory. To disable host key checking set SSH_KNOWN_HOSTS to ignore. You can also specify the path to your own known hosts file.

Compression

The compression algorithm builtin to the software is a parallel gzip (pgzip) compressor. There is support for 3rd party compressors so long as the binary is available on the host system and is compatible with the standard gzip binary command line options (e.g. xz, bzip2, lzma, etc.)

Encryption/Signing

The PGP algorithm is used for encryption/signing. The cipher used is AES-256.

Installation

Download the latest binaries from the releases section or compile your own by:

bash
go get github.com/someone1/zfsbackup-go

The compiled binary should be in your $GOPATH/bin directory.

Usage

"Smart" Backup Options

Use the --full option to auto select the most recent snapshot on the target volume to do a full backup of:

bash
./zfsbackup send --encryptTo [email protected] --signFrom [email protected] --publicKeyRingPath pubring.gpg.asc --secretKeyRingPath secring.gpg.asc --full Tank/Dataset gs://backup-bucket-target,s3://another-backup-target

Use the --increment option to auto select the most recent snapshot on the target volume to do an incremental snapshot of the most recent snapshot found in the target destination:

bash
./zfsbackup send --encryptTo [email protected] --signFrom [email protected] --publicKeyRingPath pubring.gpg.asc --secretKeyRingPath secring.gpg.asc --increment Tank/Dataset gs://backup-bucket-target,s3://another-backup-target

Use the --fullIfOlderThan option to auto select the most recent snapshot on the target volume to do an incremental snapshot of the most recent snapshot found in the target destination, unless the last full backup is older than the provided duration, in which case do a full backup:

bash
./zfsbackup send --encryptTo [email protected] --signFrom [email protected] --publicKeyRingPath pubring.gpg.asc --secretKeyRingPath secring.gpg.asc --fullIfOlderThan 720h Tank/Dataset gs://backup-bucket-target,s3://another-backup-target

"Smart" Restore Options

Add the --auto option to automatically restore to the snapshot if one is given, or detect the latest snapshot for the filesystem/volume given and restore to that. It will figure out which snapshots are missing from the local_volume and select them all to restore to get to the desired snapshot. Note: snapshot comparisons work using the name of the snapshot, if you restored a snapshot to a different name, this application won't think it is available and it will break the restore process.

Auto-detect latest snapshot:

bash
./zfsbackup receive --encryptTo [email protected] --signFrom [email protected] --publicKeyRingPath pubring.gpg.asc --secretKeyRingPath secring.gpg.asc --auto -d Tank/Dataset gs://backup-bucket-target Tank

Auto restore to snapshot provided:

bash
./zfsbackup receive --encryptTo [email protected] --signFrom [email protected] --publicKeyRingPath pubring.gpg.asc --secretKeyRingPath secring.gpg.asc --auto -d Tank/Dataset@snapshot-20170201 gs://backup-bucket-target Tank

Manual Options

Full backup example:

bash
./zfsbackup send --encryptTo [email protected] --signFrom [email protected] --publicKeyRingPath pubring.gpg.asc --secretKeyRingPath secring.gpg.asc Tank/Dataset@snapshot-20170101 gs://backup-bucket-target

Incremental backup example:

bash
./zfsbackup send --encryptTo [email protected] --signFrom [email protected] --publicKeyRingPath pubring.gpg.asc --secretKeyRingPath secring.gpg.asc -i Tank/Dataset@snapshot-20170101 Tank/Dataset@snapshot-20170201 gs://backup-bucket-target,s3://another-backup-target

Full restore example:

bash
./zfsbackup receive --encryptTo [email protected] --signFrom [email protected] --publicKeyRingPath pubring.gpg.asc --secretKeyRingPath secring.gpg.asc -d Tank/Dataset@snapshot-20170201 gs://backup-bucket-target Tank

Incremental restore example:

bash
./zfsbackup receive --encryptTo [email protected] --signFrom [email protected] --publicKeyRingPath pubring.gpg.asc --secretKeyRingPath secring.gpg.asc -d -F -i Tank/Dataset@snapshot-20170101 Tank/Dataset@snapshot-20170201 gs://backup-bucket-target Tank

Notes:

  • Create keyring files:
bash
gpg2 --gen-key
gpg2 --output public.pgp --armor --export test@example.com
gpg2 --output private.pgp --armor --export-secret-key test@example.com
  • PGP Passphrase will be prompted during execution if it is not found in the PGP_PASSPHRASE environmental variable.
  • --maxFileBuffer=0 will disable parallel uploading for some backends, multiple destinations, and upload hash verification but will use virtually no disk space.
  • For S3: Specify Standard/Bulk/Expedited in the AWS_S3_GLACIER_RESTORE_TIER environmental variable to change Glacier restore option (default: Bulk)
  • A duration string is a possibly signed sequence of decimal numbers, each with optional fraction and a unit suffix, such as "300ms", "-1.5h" or "2h45m". Valid time units are "ns", "us" (or "µs"), "ms", "s", "m", "h".

Help Output:

…

Send Options:

…

TODOs

  • Make PGP cipher configurable.
  • Refactor
  • Test Coverage
  • Add more backends
  • Add delete feature
  • Appease linters
  • Track intermediary snaps as part of backup jobs
  • Parity archives?

Issues· 0 开放

查看全部 Issues在 GitHub 打开

暂无开放 Issues,或尚未同步最近议题。

> 标签

Gobackupcloud-storagegolangzfs

暂无评论,来聊聊你的看法吧

> 工具信息

发布日期2026年8月1日
最后更新2026年9月17日
分类编程语言
定价开源

> 相关工具

T
TypeScript
JavaScript 的超集,为前端与全栈提供静态类型
P
Python
通用编程语言,广泛用于 Web、数据与 AI
G
Go
Google 推出的简洁高效系统语言