Bound `ChainState` partition growth by archiving pre-checkpoint history to object storage

Problem

Each microchain's ChainStateView is persisted as a single storage partition (RootKey::ChainState(chain_id); on ScyllaDB this is one partition of kv.table_linera, PRIMARY KEY (root_key, k)). Every view leaf (each MapView entry, LogView element, collection sub-entry) is one row in that partition, and several protocol-maintained sub-structures grow without bound and are never pruned:

• received_log: LogView<ChainAndHeight> (linera-chain/src/chain.rs) — +1 row per received cross-chain message bundle, append-only.
• block_hashes: CustomMapView<BlockHeight, CryptoHash> (post-#6163; previously confirmed_log) — +1 row per block, insert-only.

On busy chains this drives partitions past ScyllaDB's large-partition warning thresholds (compaction_rows_count_warning_threshold = 100_000 rows; compaction_large_partition_warning_threshold_mb = 1000 — both defaults on our 6.2.3 clusters, no overrides in scylla-config), causing sustained "Writing large partition" warnings, ~13x LCS write amplification on repeatedly-rewritten partitions, and degraded cache/compaction behavior. Note (ops): both violations — bytes or rows — emit the same Writing large partition …: (N bytes) log line (db/large_data_handler.cc, try_record prints an empty key field for partitions), so a 30 MB warning can be a >100k-row violation; the offending partition key is only recorded in system.large_partitions (30-day TTL).

ChainState is the largest offender but not the only partition type with unbounded per-partition growth. Three sibling per-chain partitions grow one row per block/event and are never deleted anywhere in the storage layer (written in write_confirmed_block / add_event, linera-storage/src/db_storage.rs):

• RootKey::BlockByHeight(chain_id) — height→hash certificate index (#5233): +1 row per block, the same growth rate as block_hashes above. Pruning block_hashes inside ChainState (#6599) does not touch it, so a busy chain keeps tripping the row-count threshold on this root key even after #6599.
• RootKey::Event(chain_id) — event payloads: +1 row per published event. Checkpoint summarization (#6493) removes the read guarantee below the stream floor, but no code deletes the rows.
• RootKey::EventBlockHeight(chain_id) — event→height index: +1 row per published event.

Unlike block_hashes' big-endian keys, these are BCS little-endian (to_height_key / to_event_key), so key order ≠ height order and range deletion needs key enumeration or an encoding change (same caveat as received_log below).

Measured data (production, testnet_conway)

Per-structure row counts for one large ChainState partition (a retired prediction-market chain), measured 2026-06-09/10 directly on a validator's ScyllaDB via single-partition clustering-range COUNT(*) (validator image = testnet_conway commit 84adeb0, whose view key layout is 0x01 + u32-LE field index per level):

structure	rows	share	needed after a checkpoint at tip?
received_log	8,332,095	55%	no
application KV state (execution_state.users)	4,336,259	29%	yes (live state, app-owned)
confirmed_log (= block_hashes on main)	2,402,433	16%	no
system execution state + misc	~30	~0%	yes
total	~15.07M

So ~71% of the partition (~10.7M rows) is protocol history that no node needs once a checkpoint exists above it. In a 24h sample, warning volume per GKE validator ranged from ~100 events (one validator with few but very large partitions) to ~22k events; on the OVH validator single partitions have reached 2–3 GB, crossing the byte-size threshold as well. (The application-owned 29% is tracked separately as an application data-model concern; it is out of scope here.)

Why checkpoints make this tractable

Checkpoint bootstrap already removes the need for pre-checkpoint history during sync:

• bootstrap_chain_from_checkpoint (linera-core/src/client/mod.rs) lets a node reach the producer's exact state hash by downloading only the checkpoint certificate, its CheckpointExecutionState blob(s), and the chain's referenced blobs (OracleResponse::Checkpoint, #6364). Verified end-to-end by test_checkpoint_bootstrap across memory / storage-service / RocksDB.
• #6399 (merged) lifted the no-messages restriction, and #6493 (merged) lifted the event restriction for user streams: at each checkpoint they are summarized (the app emits an absolute-state summary event; previous_event_blocks is cleared) and pre-checkpoint events lose their availability guarantee. Only system event streams remain blocking: a chain that published or consumed system events (e.g. the admin chain's epoch streams) cannot checkpoint (prepare_checkpoint / check_checkpoint_preconditions reject on system streams).

Because a node syncing past a checkpoint at height H never reads pre-H received_log / block_hashes entries, those rows can be archived to object storage (GCS/S3) and deleted from primary storage, keeping each ChainState partition bounded by (current execution state + history since the last checkpoint) — regardless of chain age. Restores from the archive should be rare and may be slow; the application/operator layer owns the archive, not the database.

Proposal (sketch — needs discussion)

1. Auto-checkpointing policy: checkpoint each chain periodically (e.g. every N months), skipping chains with no activity since their last checkpoint. The most recent blocks always stay in primary storage; only history below the latest checkpoint is ever archived.
2. Range-prune the historical sub-structures below H (selective row deletion inside the ChainState partition — the partition itself obviously stays, it holds the live state):
   • block_hashes[height < H]: keys are big-endian heights, so a range delete is expressible today via CustomMapView::remove per height or a bounded scan.
   • received_log[index < N] (N = first index at height ≥ H): requires a new LogView prefix/range-delete primitive — today LogView only has push/clear (all-or-nothing), and its BCS-u32 little-endian index keys are not lexicographically range-scannable. The stored_count/front bookkeeping must be updated accordingly.
   • the sibling per-chain partitions (see Problem above): BlockByHeight[height < H] minus the checkpoint's vouched-for blocks, and Event/EventBlockHeight below each stream's summarization floor — justified by the same checkpoint argument.
3. Archive before delete: export the pruned rows (and optionally the corresponding pre-H block/certificate/event partitions, which are separate root_keys and cleaner to handle) to object storage in a durable format.
4. Keep durable: the checkpoint certificate at H and its CheckpointExecutionState blobs (mirror them to the archive bucket as well), and all blobs referenced by used_blobs — blobs are never dropped under this proposal.

Open questions

• Bootstrap authoritativeness: latest_checkpoint_height is self-advertised, and clients silently fall back to full-history sync when a checkpoint cert can't be served. Deleting history requires a guarantee that every (re)joining node can bootstrap from a durable checkpoint. What makes checkpoint availability authoritative?
• System-event preconditions: user event streams are handled since #6493 (summarize + drop guarantee), but chains that published or consumed system events still cannot checkpoint — notably the admin chain never can. Does auto-checkpointing need an equivalent of summarization for system streams, or an explicit carve-out for these chains?
• Where the archival/GC job runs (per-validator vs coordinated), the archive format, and how a restore is triggered/served.
• Should the redundant height→hash duplication between block_hashes (inside ChainState) and the separate RootKey::BlockByHeight(chain_id) index be collapsed as part of this work?
• Checkpoint cost: dump_content re-reads and re-writes the entire execution state; for chains with millions of live app-KV rows this is significant and untested at that scale. Does auto-checkpointing need chunked/incremental dumping?