Managed activation is blocked forever by expired cycle locks: writer_not_quiesced counts any gbrain_cycle_locks row, and extract-* locks have no release path
Summary
sources writer activate (including --dry-run) treats any row in gbrain_cycle_locks as "not quiesced", regardless of ttl_expires_at or whether the holder PID is still alive. Combined with the fact that stranded cycle-lock rows have no supported release path, a single orphan row from weeks earlier permanently blocks managed activation.
The quiesce check in the activation transaction is unconditional:
await tx.executeRaw("LOCK TABLE gbrain_cycle_locks IN SHARE MODE");
if ((await tx.executeRaw("SELECT id FROM gbrain_cycle_locks LIMIT 1")).length)
throw quiescence();The very next check, for in-flight persistence work, is state-aware:
SELECT id FROM persistence_requests WHERE state IN ('queued','running','recovering') OR recovery IS NOT NULL LIMIT 1So the lock probe is the only one that cannot distinguish live work from a tombstone.
Observed
Preparing a real 0.50.5 → 0.51 managed cutover on Postgres 17:
- Every filesystem source claimed successfully (
persistence_source_bindingspopulated,persistence_worktrees.state = active,owner_epoch = 1). persistence_requestsblocking states: 0.persistence_effectswithrecovery: 0. Job queue: 0 non-terminal. No GBrain process running on any host.- Two rows remained in
gbrain_cycle_locks, bothextract-conversation-facts:default:conversations/sessions/<redacted>,ttl_expires_at13 days in the past,holder_host= this machine,holder_piddead (verified withps).
$ gbrain sources writer activate --confirm-quiesced --dry-run
Error [writer_not_quiesced]: Managed activation requires all older writers and maintenance jobs to be stopped.
Fix: Upgrade and quiesce every host, inspect existing locks, then run sources writer activate --confirm-quiesced.The error is accurate about its own rule and useless as guidance: there is nothing left to quiesce, and the "inspect existing locks" step has no follow-up command.
Why there is no way out
gbrain sync --break-lockis scoped togbrain-sync:<source>keys. It cannot seeextract-conversation-facts:*rows, and it is not listed ingbrain sync --help.- There is no
gbrain lock list/gbrain lock release. - TTL takeover only fires when the same lock id is re-acquired (
INSERT ... ON CONFLICT ... WHERE gbrain_cycle_locks.ttl_expires_at < NOW()). A lock id tied to a page whose extraction is disabled is never re-acquired, so the row never ages out.
This exact gap was reported in #2111 and closed as "a request for new CLI surface rather than a defect", with DELETE FROM gbrain_cycle_locks ... as the suggested workaround. #1470 and #4309 cover how rows get stranded in the first place.
That triage was defensible while the consequence was a stale_locks WARN in doctor. In 0.51 the same rows block a one-way upgrade, and the only documented remedy is hand-written SQL against a table the activation path locks — which is exactly the kind of step an operator should not be taking on the way into managed mode.
Minimal reproduction
- Postgres brain on 0.51.0.0.
- Strand a cycle lock: start a cycle that takes one (e.g. fact extraction over a page) and kill the process before it releases — per #1470 the release error is swallowed and the row stays. A source rename (#2111) strands one the same way.
- Wait past
ttl_expires_at, or observe the row: expired,holder_pidnot alive. - Claim every filesystem source:
gbrain sources writer claim <source> --path <root>. gbrain sources writer activate --confirm-quiesced --dry-run.
Expected: the dry-run reports readiness. Actual: writer_not_quiesced, permanently, with no supported way to clear the row.
Acceptance criteria
- A cycle lock whose
ttl_expires_athas passed and whose holder is not alive can be safely cleaned or ignored before activation, without hand-written SQL. - Either the quiesce probe ignores such rows (e.g.
WHERE ttl_expires_at > now(), optionally plus a same-host liveness check), or a supported release path exists and thewriter_not_quiescedmessage names it. - A live cycle lock still blocks activation. Nothing here should let an operator steal a lock from a running holder — the existing dead-PID/age safety checks in
runBreakLockare the right bar.
Environment
0.51.0.0, Postgres 17 behind a transaction-mode pooler, managed activation not yet enabled, filesystem sources claimed, single host.
Source: garrytan/gbrain