#1887·codegraph

Interrupted bulk-load recovery can be repeatedly watchdog-killed while rebuilding indexes

Author: cbeaulieu-gtCreated Sep 16, 2026Updated Sep 16, 2026

Summary

Opening a database after an interrupted bulk load rebuilds every missing secondary index synchronously on the main thread. If one CREATE INDEX exceeds the liveness-watchdog window, startup is killed before that index commits and every restart repeats the same recovery failure.

Context

Issue #1556 and PR #1562 added open-time restoration of indexes left missing by an interrupted bulk load. The recovery is correct, but healBulkSecondaryIndexes() executes the complete DDL sequence synchronously.

PR #1323 previously measured four edge-index builds taking 79 seconds and made the normal bulk-load completion path watchdog-safe. The crash-recovery path still performs parse, reference, and edge index recreation without yielding or moving the work off-thread.

Disk-progress deferral is insufficient when an individual CREATE INDEX produces no committed DB/WAL change before the timeout. The watchdog's hard cap can also terminate longer recovery despite intermediate disk activity.

A scaled reproduction using the real watchdog:

  1. Populate 300,000 nodes and leave the bulk-parse indexes dropped.
  2. Start recovery with the database and WAL configured as progress paths.
  3. Set CODEGRAPH_WATCHDOG_TIMEOUT_MS=20.
  4. Call DatabaseConnection.open().

Two consecutive processes were killed during the same recovery path. This uses the repository's established lowered-timeout technique for validating watchdog failure modes.

Acceptance Criteria

  • Interrupted bulk-load recovery restores every required secondary index.
  • Watchdog-supervised asynchronous opens keep the event loop responsive throughout index recreation.
  • The database is not exposed for normal queries until recovery completes.
  • Healthy opens retain their existing low-overhead fast path.
  • Regression coverage exercises recovery under a real lowered watchdog.
  • Existing synchronous API behavior remains compatible.
  • The fix is documented under the Unreleased changelog.

Technical Notes

The existing off-thread SQLite maintenance pattern provides precedent. The fix should preserve the watchdog threshold rather than weakening or disabling it.

Out of Scope

  • General schema or migration redesign.
  • Changing watchdog timeouts or progress caps.
  • Rebuilding otherwise healthy indexes.

Generated by Codex on behalf of @cbeaulieu-gt