#5197·gbrain

Managed activation is blocked forever by expired cycle locks: writer_not_quiesced counts any gbrain_cycle_locks row, and extract-* locks have no release path

Author: armandovargashCreated Sep 17, 2026Updated Sep 17, 2026

Summary

sources writer activate (including --dry-run) treats any row in gbrain_cycle_locks as "not quiesced", regardless of ttl_expires_at or whether the holder PID is still alive. Combined with the fact that stranded cycle-lock rows have no supported release path, a single orphan row from weeks earlier permanently blocks managed activation.

The quiesce check in the activation transaction is unconditional:

javascript
await tx.executeRaw("LOCK TABLE gbrain_cycle_locks IN SHARE MODE");
if ((await tx.executeRaw("SELECT id FROM gbrain_cycle_locks LIMIT 1")).length)
  throw quiescence();

The very next check, for in-flight persistence work, is state-aware:

javascript
SELECT id FROM persistence_requests WHERE state IN ('queued','running','recovering') OR recovery IS NOT NULL LIMIT 1

So the lock probe is the only one that cannot distinguish live work from a tombstone.

Observed

Preparing a real 0.50.5 → 0.51 managed cutover on Postgres 17:

  • Every filesystem source claimed successfully (persistence_source_bindings populated, persistence_worktrees.state = active, owner_epoch = 1).
  • persistence_requests blocking states: 0. persistence_effects with recovery: 0. Job queue: 0 non-terminal. No GBrain process running on any host.
  • Two rows remained in gbrain_cycle_locks, both extract-conversation-facts:default:conversations/sessions/<redacted>, ttl_expires_at 13 days in the past, holder_host = this machine, holder_pid dead (verified with ps).
$ gbrain sources writer activate --confirm-quiesced --dry-run
Error [writer_not_quiesced]: Managed activation requires all older writers and maintenance jobs to be stopped.
Fix: Upgrade and quiesce every host, inspect existing locks, then run sources writer activate --confirm-quiesced.

The error is accurate about its own rule and useless as guidance: there is nothing left to quiesce, and the "inspect existing locks" step has no follow-up command.

Why there is no way out

  • gbrain sync --break-lock is scoped to gbrain-sync:<source> keys. It cannot see extract-conversation-facts:* rows, and it is not listed in gbrain sync --help.
  • There is no gbrain lock list / gbrain lock release.
  • TTL takeover only fires when the same lock id is re-acquired (INSERT ... ON CONFLICT ... WHERE gbrain_cycle_locks.ttl_expires_at < NOW()). A lock id tied to a page whose extraction is disabled is never re-acquired, so the row never ages out.

This exact gap was reported in #2111 and closed as "a request for new CLI surface rather than a defect", with DELETE FROM gbrain_cycle_locks ... as the suggested workaround. #1470 and #4309 cover how rows get stranded in the first place.

That triage was defensible while the consequence was a stale_locks WARN in doctor. In 0.51 the same rows block a one-way upgrade, and the only documented remedy is hand-written SQL against a table the activation path locks — which is exactly the kind of step an operator should not be taking on the way into managed mode.

Minimal reproduction

  1. Postgres brain on 0.51.0.0.
  2. Strand a cycle lock: start a cycle that takes one (e.g. fact extraction over a page) and kill the process before it releases — per #1470 the release error is swallowed and the row stays. A source rename (#2111) strands one the same way.
  3. Wait past ttl_expires_at, or observe the row: expired, holder_pid not alive.
  4. Claim every filesystem source: gbrain sources writer claim <source> --path <root>.
  5. gbrain sources writer activate --confirm-quiesced --dry-run.

Expected: the dry-run reports readiness. Actual: writer_not_quiesced, permanently, with no supported way to clear the row.

Acceptance criteria

  • A cycle lock whose ttl_expires_at has passed and whose holder is not alive can be safely cleaned or ignored before activation, without hand-written SQL.
  • Either the quiesce probe ignores such rows (e.g. WHERE ttl_expires_at > now(), optionally plus a same-host liveness check), or a supported release path exists and the writer_not_quiesced message names it.
  • A live cycle lock still blocks activation. Nothing here should let an operator steal a lock from a running holder — the existing dead-PID/age safety checks in runBreakLock are the right bar.

Environment

0.51.0.0, Postgres 17 behind a transaction-mode pooler, managed activation not yet enabled, filesystem sources claimed, single host.