Stale `table_summary_v2` checkpoint wedges the summary worker permanently once document retention passes it

Summary

On a self-hosted Postgres deployment, the table_summary_v2 checkpoint in persistence_globals stopped advancing on 2026-07-12. Document retention (working correctly, 14 day window) swept past that timestamp roughly two weeks later. Since then the summary worker has retried every ~10 seconds and failed every time with out_of_retention, because it reads from the snapshot immediately after a checkpoint that is now 60 days outside the retention window.

Nothing resets the checkpoint, so the deployment cannot recover on its own. It has survived container restarts and a backend upgrade.

Symptom

Logged every ~10 seconds, indefinitely:

ERROR common::errors: Caught out_of_retention error: Document snapshot timestamp
out of leader retention window: 1783899671726798383 < 1787896492018444761:
Your request couldn't be completed. Try again later.

The left operand never changes. The right operand advances, and is exactly document_min_snapshot_ts at the time of the error.

Evidence

persistence_globals:

Both retention pairs are healthy and current: index retention is sweeping within minutes, document retention within seconds of its 14 day boundary.

The failing read is 1783899671726798383 — exactly table_summary_v2.ts + 1ns, i.e. the snapshot immediately after the checkpoint.

table_summary_v2 is 3,155,336 bytes and its contents are stale: the value distributions it records match application data as it stood on 2026-07-12.

Expected

The summary worker should either advance its checkpoint, or detect that its checkpoint has fallen outside the document retention window and re-bootstrap from a current snapshot.

Actual

It retries the same unreachable snapshot forever. The documents it needs were correctly deleted by document retention weeks earlier, so no retention setting is wide enough to recover it — the data is gone.

Environment

• Self-hosted, Docker Compose, Postgres 16.15 backing store
• ghcr.io/get-convex/convex-backend@sha256:71acf855388a71ad2fc597cd1dbea3fd262e179a07f93e754842a522024f1034
• compiled_revision 849a6fc605b5a910c9834b61a77c5475cc24d189, commit_timestamp 2026-09-05T23:56:45Z
• migration_version 129
• Document retention window observed at 14 days; DOCUMENT_RETENTION_DELAY not set explicitly
• Predates the current build and survived the upgrade to it, plus several full container restarts
• Database is small: documents 1160 MB, indexes 360 MB

Impact

Low but permanent: continuous error logging, one background worker that has not completed since July, and table summaries frozen at two-month-old values. No storage growth, no effect on queries or mutations.

Question

Is deleting the table_summary_v2 row from persistence_globals the supported recovery, so the worker re-bootstraps from a current snapshot? If so, it would be useful for the worker to do that automatically when it detects its own checkpoint is outside the retention window.

Reproducing

Not reproduced from a clean state. The trigger for the checkpoint stalling on 2026-07-12 is unknown — there is no corresponding error in retained logs, and the deployment was upgraded since. Happy to run further queries against this deployment.