A live daemon that is not holding the version-cohort marker poisons every index worker it forks, for its entire lifetime — no self-heal, and the conflict log names a build that does not exist

Summary

If a daemon is alive but is not holding the VERSION_COHORT_DAEMON_FILE marker lock, every index worker that same daemon forks rejects itself at startup and exits 1:

CBM index worker could not start: active daemon coordination could not be verified safely

The condition never clears on its own. The daemon keeps running, keeps forking workers, and every one of them fails the same way until the daemon is killed by hand. On 2026-09-11 this produced 370 conflict records over 427 minutes (~52/hour) on my machine, and it stopped at the exact second I killed the daemon.

This is why the workers fail. #2015 is about the cadence at which the watcher re-forks them once they do. They are independent: I am running the #2075 backoff, it worked, and the wedge still produced 370 dead workers — see Impact #3.

Mechanism

version_cohort_active_daemon_presence() (src/daemon/version_cohort.c:805-839) classifies via the daemon marker lock:

The worker then refuses in src/main.c:2787-2798:

cbm_version_cohort_daemon_presence_t worker_daemon_presence =
    cbm_version_cohort_daemon_presence_under_transition(worker_cohort_manager,
                                                        worker_endpoint, worker_transition);
if (worker_daemon_presence != CBM_VERSION_COHORT_DAEMON_ABSENT &&
    worker_daemon_presence != CBM_VERSION_COHORT_DAEMON_COORDINATED) {
    if (worker_daemon_presence == CBM_VERSION_COHORT_DAEMON_UNCOORDINATED) {
        (void)cbm_version_cohort_log_uncoordinated_daemon(&identity);
    }
    (void)fprintf(stderr, "CBM index worker could not start: active daemon coordination "
                          "could not be verified safely\n");
    goto worker_cleanup;
}

So the bottom row is the wedge: a daemon is alive, and it is not in the cohort. Since that daemon is also the thing forking the workers, it is rejecting its own children — and because nothing makes the daemon re-acquire the marker, the state is absorbing.

The conflict log actively misleads

cbm_version_cohort_log_uncoordinated_daemon() (src/daemon/version_cohort.c:963-980) writes a hardcoded sentinel for the active side:

(void)snprintf(conflict.active_version, sizeof(conflict.active_version), "%s", "pre-cohort/unknown");
memset(conflict.active_build_fingerprint, '0', sizeof(conflict.active_build_fingerprint) - 1);

which lands in daemon-conflicts.ndjson as:

{"event":"daemon.version_conflict","timestamp_unix_s":1789149612,"reason":"build",
 "active_version":"pre-cohort/unknown",
 "active_build":"0000000000000000000000000000000000000000000000000000000000000000",
 "requested_version":"dev",
 "requested_build":"273d3ffe8585bb5f98bb5ed99abc1be6fece06fbe50f7fda849450d216ac4e99"}

The name asserts a diagnosis — an old pre-cohort binary is running — that is not true here. The only runnable CBM binary on this machine is the 273d3ffe build in the requested_build field. (There is one other copy on disk, a March nix build in a repo working tree, but it aborts in dyld on launch and contains no watcher log keys at all, so it cannot have been the daemon.) The active daemon was almost certainly the same build as the requester — it just wasn't holding the marker.

Reporting a zero fingerprint and a version string that no binary reports sends you looking for a stale install that isn't there. A record saying "a live daemon holds no cohort marker; identity unknown" would point at the actual fault.

Impact / diagnosability

1. No self-heal. Only kill <daemon-pid> clears it. Nothing in the daemon log names the condition; index_repository and index_status returned status=error throughout without surfacing the reason.
2. The reason is invisible from the daemon side. The daemon log records only:
   复制

   level=info msg=index.supervisor.reap outcome=exit_nonzero exit_code=1 signal=0
   level=warn msg=index.supervisor.worker_failed outcome=exit_nonzero exit_code=1 log=…/.worker-log-y4KhDv

   The actual sentence is written to a per-worker .worker-log-XXXXXX temp file, never promoted. You have to know to go read a randomly-named dotfile in the log directory.
3. Backoff bounds the rate but not the total. This machine is running the #2075 branch, so the rc < 0 backoff was active throughout and behaved exactly as designed — retries decayed to the INDEX_FAIL_CEILING_MS ceiling of one per 5 min per project, and watcher.index.sustained_failure fired at consecutive=10 for six projects. Over a 427-minute wedge that ceiling still permits ~85 retries per project; liverpool-cleanup reached consecutive=81, i.e. it sat at the cap the whole time. Because the wedge never clears, dead workers accumulate linearly with daemon uptime no matter how good the backoff is — 370 conflicts across 8 watched projects in 7 hours. Capping the cadence was the right fix for #2015 and is not a fix for this.
4. Those worker-log files are never pruned — 2618 of them here going back to 2026-09-01, every one a startup failure. Minor, but it is how I found the pattern.

Field data

~/.cache/codebase-memory-mcp/logs/daemon-conflicts.ndjson, 387 records, 386 of a single shape (pre-cohort/unknown / all-zero → dev/273d3ffe):

The single remaining record is a genuine mixed-build conflict of the normal kind (version | active=dev/273d3ffe → requested=0.10.8/2412e017), which is the mechanism working as designed and is not what this issue is about.

The same cbm-daemon.log happens to contain a clean before/after for #2075, because the watcher.index.err line only carries rc / consecutive on the patched build:

There is a sibling arm one branch earlier, src/main.c:2778-2786 (cbm_daemon_ipc_local_transition_seal_legacy() != 1), which prints "a pre-coordination or unverified CBM generation is active". It produced a 2233-worker wave on 2026-09-01 — the same one quoted in #2015 — and also calls cbm_version_cohort_log_uncoordinated_daemon(). If these share a cause, both error strings should probably move with the fix.

Environment

• macOS 15 (Darwin 25.6.0), arm64
• codebase-memory-mcp dev, build 273d3ffe8585bb5f98bb5ed99abc1be6fece06fbe50f7fda849450d216ac4e99 (local build of main at v0.10.8, plus the #2075 branch)
• Several concurrent MCP frontend clients (long-lived Claude Code sessions), watcher active on ~8 projects

Repro

I do not have a deterministic repro — it has appeared three times in eight days with no action on my part that I can correlate. What I can say is that it is fully observable after the fact, and that kill <daemon-pid> is a reliable recovery. If it would help, I am happy to run an instrumented build that logs the marker-lock acquire/release path on the daemon side and report back the next time it trips.

Questions for maintainers

1. Is the daemon expected to hold VERSION_COHORT_DAEMON_FILE for its whole lifetime? If so, is there a path that releases it while the daemon stays up — and should a daemon that finds itself outside the cohort re-acquire, or fail loudly and exit rather than keep serving?
2. Should a worker forked by a daemon consult the marker at all, versus inheriting a coordination token from the parent that forked it? The current check treats the parent as an untrusted stranger.
3. Would you take a PR that (a) replaces the pre-cohort/unknown sentinel with a record stating what was actually observed, and (b) promotes the worker's failure sentence into the daemon log so worker_failed carries the reason rather than a path to a temp file? Those are small and independent of whatever fixes the underlying coordination loss.