[Bug] Codex native startup cannot recover an orphaned backfill lease, even with the 60s readiness fix

Description

A Codex native session can remain unlaunchable after an interrupted state-database backfill leaves an unexpired running lease in its private CODEX_HOME. A subsequent app-server waits 30 seconds and exits before binding its listener. Retrying startup does not reclaim the lease until it expires.

This is a recovery follow-up to #7791 / merged #7792, not a request to repeat the 10→60-second timeout fix. The original incident ran older Omnigent code with a 10-second readiness budget; current upstream already uses 60 seconds. A real Codex 0.155.1 reproduction confirms that increasing the outer budget alone cannot recover an orphaned, unexpired backfill lease: Codex itself exits after about 30 seconds.

Observed incident and evidence

Sanitized local macOS runner timeline on 2026-09-19 (UTC):

• 16:42:37: the optional codex debug models probe reported its 3-second timeout.
• 16:42:49: the private state database's persisted backfill updated_at timestamp.
• 16:42:50.994: an earlier native terminal launch failed with the 10-second app-server connection timeout and empty captured stderr.
• 16:45:37.476: the subsequent launch refreshed a resume rollout from server history (138 items).
• 16:45:47.697: that launch failed with connection refused and this app-server stderr:

state db backfill is running at <private CODEX_HOME>; waiting up to 30s before retrying startup initialization

A read-only inspection of the affected state_5.sqlite found:

backfill_state:
  id = 1
  status = running
  last_watermark = NULL
  last_success_at = NULL
  updated_at = 1789836169  # 2026-09-19T16:42:49Z

The record predates the reported retry by about three minutes, within Codex's 900-second lease. This is consistent with the earlier startup being terminated during backfill. The exact original lease owner and termination point were not captured, so that initiating sequence is a supported inference, not a proven process trace. Host contention is suggested by other timeouts but was not measured. No evidence establishes database corruption or a provider-authentication failure as the cause.

Root cause analysis

1. In the incident checkout, CodexNativeAppServer._wait_until_ready() uses _CONNECT_TIMEOUT_SECONDS = 10.0. When it raises, start() calls close(), which terminates the owned subprocess tree. The runner cannot reach terminal attachment/thread resume because _auto_create_codex_terminal() first awaits app_server.start().
2. Codex 0.155.1 persists backfill ownership in SQLite. BACKFILL_LEASE_SECONDS = 900 means another startup cannot claim a running lease until its timestamp is at least 15 minutes old. try_claim_backfill tests status and timestamp; this claim has no owner-PID liveness check.
3. wait_for_backfill_gate polls the state, attempts backfill, and returns an error after 30 seconds if it remains incomplete. With a dead lease owner and a recent timestamp, the process exits before the listener is available.
4. The incident's 10-second Omnigent timeout masked that eventual Codex startup error. #7792 fixes premature termination of slow-but-healthy startup, but does not recover this already-orphaned lease: a 60-second outer wait observes a Codex exit at ~30 seconds instead.
5. The 10-second Omnigent value, 30-second Codex gate, and 900-second Codex lease are source constants in the examined versions. No existing Omnigent config/env override controls its readiness budget. MCP startup_timeout_sec is a different setting.

Controlled reproduction

Ran the actual installed codex-cli 0.155.1 with an isolated temporary CODEX_HOME, no copied credentials/history, and real WebSocket initialize handshakes. No inference requests were made. Only fixture-owned subprocesses and the fixture database were touched.

The injected state models interrupted backfill; it does not claim to reproduce the original interruption timing. The harness allowed 40 seconds, beyond Codex's own 30-second timeout. Extending it to 60 cannot prevent an already-observed exit.

Run the following in an Omnigent development environment with codex installed (adjust the binary path to your installation):

import asyncio, os, sqlite3, tempfile, time, socket, json
from pathlib import Path
from omnigent.harnesses.codex_native.app_server import CodexAppServerClient

async def launch(home,label,budget=40):
    with socket.socket() as s:
        s.bind(('127.0.0.1',0)); port=s.getsockname()[1]
    url=f'ws://127.0.0.1:{port}'
    started=time.monotonic()
    proc=await asyncio.create_subprocess_exec('codex','app-server','--listen',url,env={**os.environ,'CODEX_HOME':str(home)},stdin=asyncio.subprocess.DEVNULL,stdout=asyncio.subprocess.DEVNULL,stderr=asyncio.subprocess.PIPE)
    outcome='deadline'
    try:
        while time.monotonic()-started < budget:
            if proc.returncode is not None:
                outcome=f'exit {proc.returncode}';break
            client=CodexAppServerClient(ws_url=url)
            try:
                await asyncio.wait_for(client.connect(),2)
                outcome='ready'; break
            except Exception: pass
            finally: await client.close()
            await asyncio.sleep(.1)
    finally:
        elapsed=time.monotonic()-started
        if proc.returncode is None: proc.terminate()
        _,err=await asyncio.wait_for(proc.communicate(),5)
        print(json.dumps({'case':label,'outcome':outcome,'seconds':round(elapsed,3),'stderr':err.decode()[-2500:]}),flush=True)
    return outcome

async def main():
    with tempfile.TemporaryDirectory(prefix='codex-backfill-rca-') as d:
        h=Path(d)
        assert await launch(h,'fresh')=='ready'
        db=next(h.glob('state_*.sqlite'))
        def mark(age):
            with sqlite3.connect(db) as c:
                c.execute("update backfill_state set status='running',last_watermark=NULL,last_success_at=NULL,updated_at=? where id=1",(int(time.time())-age,))
        mark(0)
        await launch(h,'orphaned-unexpired-lease')
        mark(901)
        await launch(h,'expired-lease')
asyncio.run(main())

Expected behavior / proposed scope

• Recognize and surface the backfill-blocked startup distinctly, including actionable, non-destructive recovery guidance.
• Determine whether recovery belongs in Codex lease ownership/liveness handling or in a supported Omnigent retry/recovery path; the 15-minute lease mechanics are vendor-owned.
• Preserve resumable session history and runtime state. Do not blindly delete the database or reset a lease that may belong to a live worker.
• Validate both a live concurrent backfill owner and a dead owner; retain prompt failure for unrelated crashes.
• Treat #7792 as the existing slow-start mitigation, not as proof that orphaned-state recovery is solved.

Version

Incident checkout: cb495aa49d1bb74d5dad3f19726e1651ad7cba25 (local production branch). RCA also inspected upstream main 0da52affceed86107a5cf7587ec0d8d8e73a5a5d, which contains #7792 / 4a0dce07bb8a133a4e6db4f82cb8138fa34b2040.

Installed binary used for the controlled reproduction: codex-cli 0.155.1. The original runner log does not record the binary version, so exact incident-version equivalence is not independently established.

OS / Harness / Mode / Platform

macOS 26.6.2 (25G83), Apple Silicon; Codex, native (codex-native), local runner and loopback WebSocket app-server.

Observed impact

One observed resumed session with repeated launch failures; broader incidence is unknown. Controlled reproduction establishes that an orphaned lease can block startup until lease expiry even after adopting the larger Omnigent readiness budget.

Authentication type

Not authentication-related in the reproduction. The incident also logged provider fallback, but failure occurred before the app-server listener was available; no model authentication exchange was reached in this failing launch.

Related work / validation limits

• #7791 / #7792: original 10-second readiness defect, already fixed upstream.
• #6722: model-options polling before listener readiness, a different caller.
• #6981: interactive TUI prompts after app-server startup, a different phase.
• openai/codex#35674: related backfill-blocked startup with large history; this reproduction needs no large history, only an orphaned unexpired lease.

A local 60-second readiness patch passed 140 app-server unit tests, but those mocked delayed-listener tests do not prove orphaned-lease recovery. The real-process test above establishes the remaining limitation. The affected user's database was inspected read-only and was not repaired or deleted during RCA.