Cross-agent `issue_comment_mentioned` wake for a non-assignee target strands in `deferred_issue_execution` (recovery sweep filters by assignee only)

Summary

A cross-agent issue_comment_mentioned wake queued while the issue's execution lock was held stays in deferred_issue_execution indefinitely when its target agent is not the issue's current assignee, unless some later assignee-scoped activity drives the issue's releaseIssueExecutionAndPromote path. In our production instance the same ticket has now produced this deadlock twice: 31d 7h on wake d5b98bc0-… (measured 2026-08-24) and 24d 0h on a second independent wake ea123f73-… on the same ticket (measured 2026-09-17). The stranded-queue recovery sweep that is supposed to unblock this class filters its join by assigneeAgentId = wake.agentId, so non-assignee mention wakes are structurally excluded from recovery.

Reproduced on [email protected] (latest master, released 2026-09-16 18:06 UTC).

Mechanism (current bundle: @paperclipai/server/dist/services/heartbeat.js)

1. Agent A holds the issue lock (issues.executionRunId != null). Agent A posts a comment that mentions agent B. enqueueWakeup parks B's issue_comment_mentioned wake as deferred_issue_execution because the lock is busy (by design). The wake's agentId = B; the issue's assigneeAgentId = A.

2. A's run finalizes. releaseIssueExecutionAndPromote calls into wakeQueue.releaseIssueExecution → runReleaseDrain. runReleaseDrain iterates findNextDeferredWake per-issue and would happily promote B's wake — but only if the finalizing run actually holds the issue's execution lock at the moment withIssueExecutionLock runs.

3. If A's lock was already released for any other reason before A's finalization reached the release path (finalization sequencing quirks, an interstitial orphaned_running_run sweep clearing the lock, a missing_issue_comment retry cancel path that clears without re-promoting for non-assignee targets, an interrupt-race that took the receipt off deferred_issue_execution and back), then B's wake never gets drained by any finalization path — the issue has no more incoming A-owned runs, and B never gets adopted as a candidate.

4. The recovery sweep in resumeQueuedRuns (the "stranded queues" branch) is designed to fix exactly this shape — but its join is:

   // heartbeat.js:13233-13237  ([email protected])
   const strandedQueues = await db.select({ wake: agentWakeupRequests })
       .from(agentWakeupRequests)
       .innerJoin(issues, and(
           eq(issues.companyId, agentWakeupRequests.companyId),
           sql`${issues.id}::text = ${agentWakeupRequests.payload}->>'issueId'`,
           eq(issues.assigneeAgentId, agentWakeupRequests.agentId),   // <-- excludes non-assignee wake targets
       ))
       .innerJoin(companies, and(eq(companies.id, issues.companyId), eq(companies.status, "active")))
       .where(and(
           eq(agentWakeupRequests.status, "deferred_issue_execution"),
           isNull(issues.executionRunId),
           sql`jsonb_typeof(${agentWakeupRequests.payload} #> '{_paperclipWakeContext,wakeCommentIds}') = 'array'`,
           sql`${agentWakeupRequests.payload} #> '{_paperclipWakeContext,wakeCommentIds}' <> '[]'::jsonb`,
           sql`${agentWakeupRequests.payload}->'queuedCommentInterrupt' is null`,
           cutoff ? gte(agentWakeupRequests.requestedAt, cutoff) : undefined,
       ))
       .orderBy(asc(agentWakeupRequests.updatedAt)).limit(50);
   

   The eq(issues.assigneeAgentId, agentWakeupRequests.agentId) predicate matches the assignee, not the wake target. issue_comment_mentioned wakes are inherently cross-carrier: the wake fires because the mentioned agent is not the assignee. Every such deferred wake against an idle issue is invisible to this sweep, forever.

5. Because the wake row is still deferred_issue_execution (not cancelled, not failed), it looks like a live liveness path to other health checks. Nothing pages, nothing retries. The wake is delivered only when some unrelated later event finally causes a releaseIssueExecutionAndPromote on the issue — in our data, that has taken up to 31 days.

Production evidence (this instance, self-hosted [email protected])

Two independent deferrals on the same issue (identifier LEG-1844, id 1ddf20cc-4b91-4656-9314-3e82dd542f69, assignee 474b90ac-… "SELLY"):

wake id (prefix)	wake reason	wake target agent	requested (UTC)	delivered (UTC)	wall-clock deferred
d5b98bc0-…	issue_comment_mentioned	non-assignee (BOB, 6de3da99-…)	~2026-07-24	2026-08-24	31 d 7 h 46 m
ea123f73-…	issue_comment_mentioned	non-assignee (BOB, 6de3da99-…)	2026-08-24	2026-09-17	24 d 0 h

Both wakes eventually fired only after an unrelated event finally drove a releaseIssueExecutionAndPromote on the ticket. In neither case did the stranded-queue sweep participate — the join eliminated them.

The second deferral proves this is not a one-off environmental incident: the same issue reproduced the exact class again in the following month under the latest master. Bundle version 2026.916.0 was live at measurement time (npm view paperclipai version).

Related public work

• #13374 (open PR) — prevents cross-tree exec lock on @-mention wakes. Targets the opposite direction (mention run's runId stamped as the issue's executionRunId blocking the assignee). Does not address stranded non-assignee deferred wakes when the issue is idle.
• #12671 (open PR, refresh of #11168) — recovers deferred handoffs after two specific stale-execution paths (missing-comment retry cancel with issue_assignee_changed; orphaned_running_run sweep clearing the lock). Its recovery scan explicitly reaches only "the wake's current assignee" — so it inherits the same assignee-only filter and does not cover cross-carrier mention wakes with a non-assignee target.
• #10195 (open) — different starvation mode (promotion loop stops after promoting the oldest stale wake, leaving fresh ones behind). Different mechanism.
• #10199, #11168 — earlier fixes in the finalization promotion loop; do not address recovery-sweep coverage for non-assignee targets.

Proposed fixes (in order of scope)

1. Broaden the stranded-queue recovery join to admit non-assignee wake targets. Drop or relax the eq(issues.assigneeAgentId, agentWakeupRequests.agentId) predicate at heartbeat.js:13235 (equivalent source in services/heartbeat.ts). Any deferred wake for an issue with executionRunId IS NULL and a live comment context should be re-driven through releaseIssueExecutionAndPromote — invokability, pause-hold, and self-authorship guards inside runReleaseDrain already gate what actually promotes. The current predicate is stricter than the drain that follows it, so removing it does not weaken promotion decisions; it only lets the recovery path see the class of stranded wake that is currently invisible.

2. Age-based visibility, not just recency. The current sweep uses updatedAt ordering with a bounded limit(50) and requires requestedAt >= cutoff. A wake stuck > 24 h against an idle issue should be at least surfaced (log + metric) even if not auto-promoted, so the class is measurable in the wild instead of invisible until a customer notices.

3. Watchdog / observability. Emit a warning-level log and a counter each time the sweep skips a deferred_issue_execution row older than N hours because the assignee join failed. Zero cost when the join predicate is broadened per (1); high signal today.

Steps to reproduce (integration test outline)

1. Create two agents A and B in one company; assign issue I to A.
2. Start an issue-bound run for A on I; hold the execution lock (issues.executionRunId = runA.id).
3. During that run, have A post a comment on I that mentions B — this enqueues B's issue_comment_mentioned wake, which admission parks as deferred_issue_execution (payload includes the mention's comment id).
4. Finalize A's run through a path that clears the execution lock without having the release drain reach B's wake — the cleanest reproduction is: while A's run is still active, terminate A's process out-of-band so orphaned_running_run sweep runs; the sweep clears the lock, but B's wake row is not promoted because runReleaseDrain never fires for B's finalization (A never finalizes cleanly).
5. Do nothing else on I. Run resumeQueuedRuns on a schedule.
6. Expected: B's wake is delivered promptly (bounded latency, e.g. within one sweep cycle).
7. Actual on current master (2026.916.0): B's wake stays deferred_issue_execution indefinitely. The strandedQueues sweep never selects it because issues.assigneeAgentId != B.id.

Deployment mode

Self-hosted npm distribution (npx paperclipai run), embedded Postgres, claude_local and codex_local adapters. Not adapter-specific — the fault is in core services/heartbeat.js scheduling and recovery.

Impact

For any team where one agent regularly @-mentions another agent on a ticket the mentioned agent does not own (which describes the normal CTO/CEO/reviewer collaboration pattern), a fraction of those mentions can silently vanish for weeks. Because the deferred row still looks live to health checks, no page fires. In our operation this caused a 32-day board silence documented in one internal incident and now a second independent 24-day incident on the same ticket.