Writeup: retries delete previously produced PDFs, the returned success flag is unreliable, and the final reflection can leave a corrupted template.tex that no longer matches the emitted PDF

Related issues: no direct duplicate found (searched: writeup, pdf, reflection, template.tex, retry). #16 ("How can i get the PDF?") looks like a user-facing symptom of writeup failing without artifacts, but contains no root cause analysis — possibly RELATED #16.

Summary

Three coupled defects in perform_icbinb_writeup.py (the ICBINB writeup path) combine so that a run which actually produced a good paper PDF can (a) be reported as a failure, (b) have that PDF deleted by the automatic retry, and (c) end with a latex/template.tex that does not correspond to any PDF on disk.

Where

All references are to current main (96bd516).

(a) Unreliable success flag — ai_scientist/perform_icbinb_writeup.py:1206–1242. After the numbered reflection loop, reflection_pdf is reassigned to the final-page-limit name before checking whether the model even returned LaTeX, and the return value tests only that file:

        reflection_pdf = osp.join(
            base_folder, f"{osp.basename(base_folder)}_reflection_final_page_limit.pdf"
        )
        ...
        reflection_code_match = re.search(
            r"```latex(.*?)```", reflection_response, re.DOTALL
        )
        if reflection_code_match:
            reflected_latex_code = reflection_code_match.group(1).strip()
            if reflected_latex_code != current_latex:
                ...
                with open(writeup_file, "w") as fo:
                    fo.write(final_text)

                compile_latex(latex_folder, reflection_pdf)
            else:
                print(f"No changes in reflection page step.")

        return osp.exists(reflection_pdf)

    except Exception:
        print("EXCEPTION in perform_writeup:")
        print(traceback.format_exc())
        return False

If the final reflection returns no fenced ```latex block, or returns LaTeX identical to the current source ("No changes in reflection page step." — arguably the best outcome), *_reflection_final_page_limit.pdf is never compiled and the function returns False, even though earlier *_reflection{i}.pdf files (lines 1027–1032) were successfully produced. Any late exception likewise returns False with good PDFs already on disk.

(b) Retry deletes prior artifacts — ai_scientist/perform_icbinb_writeup.py:867–879; every call begins by destroying all previous output:

    pdf_file = osp.join(base_folder, f"{osp.basename(base_folder)}.pdf")
    latex_folder = osp.join(base_folder, "latex")

    # Cleanup any previous latex folder and pdf
    if osp.exists(latex_folder):
        shutil.rmtree(latex_folder)
    if osp.exists(pdf_file):
        os.remove(pdf_file)

    # Remove any previous reflection PDFs
    for old_pdf in os.listdir(base_folder):
        if old_pdf.endswith(".pdf") and "reflection" in old_pdf:
            os.remove(osp.join(base_folder, old_pdf))

The launcher retries on a False flag (launch_scientist_bfts.py:278–300), so (a)+(b) means: attempt 1 produces a usable PDF but returns False → attempt 2 starts by deleting it → if attempt 2 fails harder (e.g. LLM/compile error), the run ends with zero PDFs despite having had one. The deletion is also a race window for any concurrent consumer reading PDFs from the idea directory while a retry starts.

(c) tex/PDF mismatch after a failed final reflection — lines 1230–1233 write the final reflected LaTeX into latex/template.tex and then compile. compile_latex (lines 45–85) swallows all pdflatex errors (prints and continues; the subprocess.run calls don't use check=True, so CalledProcessError can't even fire) and only logs when shutil.move(template.pdf, ...) finds nothing to move. So when the final reflection emits broken LaTeX, template.tex on disk is the broken version while the surviving PDFs were compiled from an earlier, good version. Observed failure modes from the model in this last step include referencing a nonexistent style name (e.g. iclr2025_conference instead of the template's actual style file) and "Lonely \item" errors.

Impact

From a multi-run reproduction campaign:

• Multiple runs logged writeup success: False while a usable reflection PDF existed in the idea directory; in those runs the retry then deleted it (recovered only because we archived artifacts between attempts).
• 2/8 writeup runs in one batch finished with a corrupted template.tex whose content could not have produced the PDFs on disk — any downstream step that assumes tex↔PDF correspondence (post-hoc edits, source audits, camera-ready regeneration) silently operates on the wrong source.

Repro sketch

1. Run the ICBINB writeup with a model that occasionally answers the final page-limit reflection without a ```latex fence (most reasoning-style models) — observe return False despite *_reflection{i}.pdf existing.
2. With --writeup-retries 2+, observe attempt 2 deleting attempt 1's PDFs at lines 871–879.
3. For (c): let the final reflection emit LaTeX with an invalid style reference; observe pdflatex failing silently inside compile_latex, template.tex containing the broken source, and the directory still holding older good PDFs.

Suggested fix (minimal)

• Success flag: return True if any reflection PDF exists (track the newest successfully compiled one), e.g. return any(f.endswith(".pdf") and "reflection" in f for f in os.listdir(base_folder)); reserve False for "no PDF was produced at all". The "no changes" branch in particular should not fail the writeup.
• Retry safety: don't delete previous PDFs at function entry. Compile to temporary names and move into place on success, or delete an old PDF only at the moment its replacement exists.
• tex/PDF consistency: make compile_latex report success (e.g. return osp.exists(pdf_file) after the move); after the final reflection compile, if it failed, restore the previous known-good template.tex (keep a copy before overwriting) so the tree on disk stays self-consistent.