fix(news): separate digest retention from forecast evidence and prune tombstones

Parent

#7080

Priority

P0 — the current structure is unbounded and shares one key between incompatible 48-hour and 7–14-day contracts.

What to build

Give digest accumulation and forecast judging separate retention contracts. The digest accumulator must stay bounded to 48 hours. Forecast judging must use a dedicated, self-contained archive that remains usable for its declared maximum lookback and does not depend on seven-day story-track rows.

Why

The current writer refreshes the accumulator key TTL on every build but never removes old sorted-set members. Production contains millions of expired members and tombstones. A direct 48-hour trim is unsafe because the forecast resolver reads the same key for up to 14 days.

Implementation plan

1. Freeze reader and data contracts

• Inventory every in-repo reader of digest accumulators and record its key scope, lookback, fields, and failure behavior.
• Define the digest contract as a 48-hour rolling index used by briefs, watchlists, and spike calculations that request no longer history.
• Define the forecast archive as a versioned, self-contained evidence record with the minimum fields required to judge a forecast. Its maximum retention must cover the 14-day reader contract plus a small cleanup guard band.
• Do not make the forecast archive depend on a story hash whose record expires after seven days.

2. Add the archive before pruning

• Add dual publication of eligible full-English evidence to the new forecast archive.
• Backfill the still-readable portion of the last 14 days before any destructive cleanup. Record unreadable tombstones and missing fields instead of silently omitting them.
• Keep archive writes bounded by explicit record, byte, and command budgets.
• Add archive version, publication time, source digest time, and retention metadata.

3. Migrate and verify the forecast reader

• Move forecast judging to the dedicated archive.
• Compare old and new reader results over the same eligible window while both paths exist.
• Fail closed when archive coverage is truncated, unreadable, or outside the declared window. Do not convert missing evidence into a judged negative.
• Remove the old accumulator dependency only after the comparison passes.

4. Enforce digest retention

• Add member-level pruning for scores older than 48 hours as part of normal digest publication.
• Keep key TTL as abandoned-key cleanup, not as member retention.
• Add a bounded cleanup tool with dry-run output. It must enumerate exact keys and report cardinality and oldest/newest score before mutation.
• Run the one-time production sweep only after the forecast cutover is deployed and verified. Capture the pre-sweep evidence needed for recovery or audit.

5. Observe steady state

• Record accumulator cardinality, oldest age, prune count, and command latency in operator monitoring.
• Keep ordinary cardinality drift out of GitHub status checks. Integrity, malformed archive records, and retention-contract violations must still fail closed where they affect judging.

Acceptance criteria

• All accumulator readers and their required lookbacks are listed in the pull-request handoff.
• Forecast judging uses a dedicated archive and no longer reads a digest accumulator.
• A forecast evidence item remains readable at day 13 and is ineligible after the declared maximum retention.
• The archive remains usable after the corresponding story-track row expires.
• A digest accumulator member older than 48 hours is removed during normal publication.
• Boundary tests keep exactly eligible members at the 48-hour edge and do not remove newer members.
• The migration reports old/new reader parity, archive truncation, missing evidence, and tombstones.
• The cleanup tool defaults to dry-run and requires explicit key scope before mutation.
• The production sweep records UTC time, key count, member counts, oldest ages, and the remaining forecast-archive coverage.
• Post-deploy observation shows bounded cardinality for at least three consecutive digest publication cycles.

Test plan

• Use a fake clock to test 48-hour digest and 14-day forecast boundaries.
• Test archive reads after story-track expiry.
• Test malformed, truncated, duplicate, and partially backfilled archive records.
• Test that a failed archive write does not advance coverage metadata.
• Test the cleanup dry-run, exact-key allowlist, and idempotent second run.
• Run focused news, forecast-resolution, Redis-cache, and story-tracking tests, followed by the API typecheck and diff hygiene checks.

Blocked by

None — can start immediately.

Out of scope

• Changing forecast scoring or judgment policy.
• Extending digest retention beyond 48 hours.
• Treating tombstones as negative forecast evidence.
• Pruning production before the new reader is deployed and verified.
• Hiding ordinary storage health inside pull-request CI.