[backup] manifest compaction

Background

The backup coordinator keeps listening to a Diem node (specifically, its backup service) and dumps Epoch ending ledger infos, transactions (and events) and state snapshots in relatively small intervals to configured backup storage (for example, a AWS EC3 bucket). Each of the individual backups has a corresponding metadata file in the metadata folder describing it. The backup format is described in this.

The restore coordinator, on the other hand, loads the entirety of the metadata folder and decides which backups should be involved to recreate the latest DiemDB.

The metadata files are plenty because we want to keep the intervals to backup everything small, so that the backup is never too much behind the blockchain. As a result loading the metadata folder takes a while.

The task

To shorten the time to load the metadata, we wish to be able to combine multiple metadata files after the fact without needing to take backups at longer intervals.

• Read the code and understand how things work around the backup system.
• Run a local diem node (cargo run -p diem-node -- --test) and try to invoke the backup coordinator (cargo run -p backup-cli --bin db-backup -- auto ..., figure out what parameters to use) to create a backup, run the db-restore tool to recover, etc. Play with it.
• Make backup compactor, as part of the backup coordinator or a stand alone tool.
  • The BackupStorage trait needs an upgrade to include our first interface that can DELETE things.
  • The restore coordinator must bare with duplicate entries that are supposed to come from different files when the compaction is in progress.
  • The tool need to have configs defined and logic around the strategy to create combined metadata files from smaller ones.
  • It creates larger metadata files before deleting smaller ones, or optionally, moving them to another folder / renaming them to *.bak.
  • (Stretch) not only the metadata can be combined, the backup themselves as well.