Two v2 bugs (F11 VLM whitelist rejection, F13 UnboundLocalError on pdf_path) reproduce in 24/24 biomedical-prompt runs at SHA 96bd51617
Evaluated SHA: 96bd51617cfdbb494a9fc283af00fe090edfae48 Primary model: DeepSeek-Chat Corpus: n=10 substantive prompts, 24 write-up attempts Pre-registration: https://osf.io/aqchj
Filing two bugs that reproduce in every one of 24 write-up attempts against the SHA above, surfaced during an independent evaluation of v2 on biomedical research prompts.
F11 — VLM whitelist rejection
The vision-language-model figure-review step rejects gemini-2.5-flash even when it's configured as the VLM. Visible in every phase3.log I captured (n=10, one per substantive prompt). The whitelist check appears to exclude currently-configured models. Behavior is fail-safe (no crash) but the VLM figure-review step is effectively skipped, so v2 never actually reviews its own figures at write-up time.
F13 — UnboundLocalError on pdf_path in write-up phase
pdf_path is referenced before assignment in the manuscript write-up module under conditions I observed in every one of 24 write-up attempts. Runtime error surfaces after the write-up completes — the PDF is still produced (the error path doesn't prevent output) but the process exits non-zero, which downstream automation may misinterpret as a failed run.
Reproducibility
Full replication package (24 output PDFs, 20 ideation records, phase3.log files, scoring scripts) will be released with the paper. If either bug is fixed in a post-SHA commit, a pointer would help so the fix can be noted in the camera-ready.
Source: SakanaAI/AI-Scientist-v2