Interrupted bulk-load recovery can be repeatedly watchdog-killed while rebuilding indexes
Summary
Opening a database after an interrupted bulk load rebuilds every missing
secondary index synchronously on the main thread. If one CREATE INDEX
exceeds the liveness-watchdog window, startup is killed before that index
commits and every restart repeats the same recovery failure.
Context
Issue #1556 and PR #1562 added open-time restoration of indexes left missing
by an interrupted bulk load. The recovery is correct, but
healBulkSecondaryIndexes() executes the complete DDL sequence synchronously.
PR #1323 previously measured four edge-index builds taking 79 seconds and made the normal bulk-load completion path watchdog-safe. The crash-recovery path still performs parse, reference, and edge index recreation without yielding or moving the work off-thread.
Disk-progress deferral is insufficient when an individual CREATE INDEX
produces no committed DB/WAL change before the timeout. The watchdog's hard cap
can also terminate longer recovery despite intermediate disk activity.
A scaled reproduction using the real watchdog:
- Populate 300,000 nodes and leave the bulk-parse indexes dropped.
- Start recovery with the database and WAL configured as progress paths.
- Set
CODEGRAPH_WATCHDOG_TIMEOUT_MS=20. - Call
DatabaseConnection.open().
Two consecutive processes were killed during the same recovery path. This uses the repository's established lowered-timeout technique for validating watchdog failure modes.
Acceptance Criteria
- Interrupted bulk-load recovery restores every required secondary index.
- Watchdog-supervised asynchronous opens keep the event loop responsive throughout index recreation.
- The database is not exposed for normal queries until recovery completes.
- Healthy opens retain their existing low-overhead fast path.
- Regression coverage exercises recovery under a real lowered watchdog.
- Existing synchronous API behavior remains compatible.
- The fix is documented under the Unreleased changelog.
Technical Notes
The existing off-thread SQLite maintenance pattern provides precedent. The fix should preserve the watchdog threshold rather than weakening or disabling it.
Out of Scope
- General schema or migration redesign.
- Changing watchdog timeouts or progress caps.
- Rebuilding otherwise healthy indexes.
Generated by Codex on behalf of @cbeaulieu-gt
Source: colbymchenry/codegraph