Stage summaries are only persisted after all 4 stages finish — interrupted runs leave complete journals but no summaries, blocking the writeup stage

Related issues: RELATED #42 ("Error in overall_summarize function") — different root cause (an exception inside overall_summarize), but it illustrates the same fragility: because all four summaries are computed and written in one shot at the very end, a single late failure loses every summary. No duplicate of the persistence-timing issue itself was found (searched: summary, stage, journal, interrupt).

Summary

Per-stage journals are checkpointed to disk after every search step, but the four stage summaries (draft/baseline/research/ablation_summary.json) — which the writeup stage hard-depends on — are computed and written only once, after manager.run() has returned, i.e. after all four BFTS stages complete. Any run killed before that point (wall-clock cap, OOM killer, operator interrupt, or one of the framework crashes reported separately) leaves fully usable journal.json files for every completed stage but zero summary files, so perform_writeup cannot run on those artifacts without offline reconstruction.

Where

Current main (96bd516).

Summaries: end-of-run only. ai_scientist/treesearch/perform_experiments_bfts_with_agentmanager.py:211–250:

    manager.run(exec_callback=create_exec_callback(status), step_callback=step_callback)
    ...
    if cfg.generate_report:
        print("Generating final report from all stages...")
        (
            draft_summary,
            baseline_summary,
            research_summary,
            ablation_summary,
        ) = overall_summarize(manager.journals.items(), cfg)
        ...
        with open(draft_summary_path, "w") as draft_file:
            json.dump(draft_summary, draft_file, indent=2)
        # (same for baseline/research/ablation, lines 240–250)

Journals: every step. The step callback saves the journal continuously — perform_experiments_bfts_with_agentmanager.py:152 calls save_run(cfg, journal, stage_name=f"stage_{stage.name}"), which writes logs/0-run/stage_*/journal.json (ai_scientist/treesearch/utils/config.py:219–227).

Hard downstream dependency. ai_scientist/perform_icbinb_writeup.py:665–672 (load_exp_summaries) reads exactly these files:

    summary_files = [
        ("logs/0-run/baseline_summary.json", "BASELINE_SUMMARY"),
        ("logs/0-run/research_summary.json", "RESEARCH_SUMMARY"),
        ("logs/0-run/ablation_summary.json", "ABLATION_SUMMARY"),
    ]

(perform_writeup.py:489–491 mirrors this for the non-ICBINB path.)

Impact

• In a batch reproduction campaign with a fixed per-run wall-clock cap, every run that hit the cap (5 of 20 in one batch) had complete, current per-stage journal.json files for all stages reached, but no summary files, making the standard writeup entry point unusable as-is.
• Recovering required re-implementing the summarization offline from journal.json (re-running annotate_history / get_stage_summary per stage). The __main__ block of ai_scientist/treesearch/log_summarization.py (lines 364–452) contains scaffolding for exactly this reconstruction, but it is hardcoded to an example path and not wired into the launcher — evidence the need is known.
• The cost asymmetry is large: a 10-hour tree search loses its entire downstream pipeline because four small JSON files (each derived from one stage's journal, independently) were deferred to the very end.

Repro sketch

1. Start launch_scientist_bfts.py on any idea.
2. Kill the process during Stage 3 or 4 (SIGKILL, or simulate a scheduler timeout).
3. Inspect logs/0-run/: stage_*/journal.json files are present and current; no *_summary.json exists.
4. Run the writeup on that idea directory: load_exp_summaries finds none of its inputs.

Suggested fix (minimal)

Persist each stage's summary as soon as that stage completes, instead of all four at the end:

• On stage transition in the agent manager (the point where a stage's journal is final), run that stage's branch of overall_summarize/get_stage_summary and write the corresponding *_summary.json immediately. The four summaries are computed independently per stage (see process_stage in log_summarization.py:302–347), so this is a re-ordering, not a redesign.
• Cheaper stopgap: wrap manager.run() in try/finally and, in the finally, run overall_summarize over whichever journals exist, writing whatever summaries can be produced.

Either variant also reduces the blast radius of #42-style failures: an exception while summarizing one stage no longer destroys the other three summaries.