A runner that exits during shutdown leaves the simulation permanently unreportable (distinct from #795)
Thanks for #798 — it fixed the failure I hit second. This is the one I hit first, and I believe it is a different bug that #798 does not cover.
The difference from #795
Both end with runner_status: stopping and a report that can never be generated, but the cause is opposite:
| #795 / #798 | this | |
|---|---|---|
| runner process | alive, hard-waiting on episode.processed |
exited |
| fix by shortening the wait | yes | no — there is no loop left to shorten |
| recovery after restart | resolves once the wait ends | never, the state is on disk |
What happened
A 40-round, 2-world run completed cleanly — both platforms reported done, 71 actions. During the shutdown wait the runner process died (in my case while Zep polling was flapping with ReadTimeout / ConnectError / RemoteProtocolError). It never wrote its terminal status.
run_state.json was left as:
{
"runner_status": "stopping",
"completed_at": null,
"current_round": 40, "total_rounds": 40,
"twitter_completed": true, "reddit_completed": true,
"process_pid": 1747 // no longer alive
}
POST /api/report/generate then refuses forever, correctly per its own gate:
409 Simulation or Zep graph ingestion is still active;
wait for a terminal run status before generating a report
because RunnerStatus.STOPPING is in active_statuses (backend/app/api/report.py).
Why it cannot recover
SimulationRunner.get_run_state() loads run_state.json as-is and never checks whether process_pid is still alive, so a restart faithfully reloads the stale non-terminal state. Nothing in the UI can clear it. The simulation's data is complete and correct, but it is permanently unreportable.
The only way out I found was editing run_state.json by hand to "runner_status": "completed", which is not something a user should have to do — and is risky, since nothing stops you writing completed over a run that genuinely did not finish.
Suggested direction
On load, reconcile a non-terminal run_state whose process_pid is not alive. The evidence needed is already in the file — if every platform reports completed and current_round == total_rounds, it finished; otherwise it is FAILED rather than STOPPING. Either way it becomes terminal and the UI stops lying about it.
A manual "mark this run finished" affordance would also work, but reconciliation on load seems better: the user should not have to know what a PID is.
Environment
- MiroFish at 39d8491 (
#792), with #798 applied locally (132 backend tests pass) - macOS, Python 3.12 via uv, Node 26
- LLM: OpenRouter (
google/gemini-3.5-flash), Zep Cloud free tier
Happy to open a PR for the reconciliation if the approach sounds right.
Source: 666ghj/MiroFish