Plot aggregator asks the LLM to guess experiment_data.npy structure → empty figure sets that 5 reflection rounds cannot fix; attach a real data-structure manifest to the prompt

Related issues: no existing issue found (searched: aggregator, plot, npy, figures, empty).

Summary

aggregate_plots builds final paper figures by asking an LLM to write a Python script that loads the runs' experiment_data.npy files. The prompt contains only the stage summaries (which include the .npy paths) — never the contents/structure of those files. In practice each experiment_data.npy is a pickled dict with run-specific, unpredictable top-level keys (one per dataset/variant/ablation/seed) and heterogeneous value shapes (scalars, lists, lists of dicts, nested dicts). The LLM therefore hardcodes guessed keys; the generated script runs to completion with exit code 0 but produces zero or blank figures, and the reflection loop — which only sees the figure count and script stdout, not the data — cannot recover.

Where

Current main (96bd516), ai_scientist/perform_plotting.py:

• build_aggregator_prompt (lines 52–86) — the only data-related guidance is path-level:

IMPORTANT:
- The aggregator script must load existing .npy experiment data from the "exp_results_npy_files" fields (ONLY using full and exact file paths in the summary JSONs) for thorough plotting.
...
4) Do not hallucinate data. Data must either be loaded from .npy files or copied from the JSON summaries.

Nothing tells the model what is inside the files, and nothing forbids hardcoding a single guessed key.

• aggregate_plots (lines 136–156) loads idea text + summaries, builds this prompt, and runs up to n_reflections: int = 5 reflection rounds (line 137). The reflection prompt (lines 203–219) feeds back only figure_count and the script's output — so when the root cause is "script iterated over a guessed key that doesn't exist / matched nothing", every reflection round repeats a variant of the same guess.

• A success log is emitted based on exit code alone ("ran successfully"), even when figures/ ends up with zero non-empty plots.

Impact

Measured in a multi-run reproduction campaign:

• One representative aggregation: the input .npy files together contained 37 distinct top-level keys (heterogeneous datasets/ablations); the generated aggregator hardcoded one guessed synthetic-dataset key, "ran successfully", and produced 0 usable figures across all 5 reflection rounds.
• After a prompt-only change — dumping the actual structure of every .npy (top-level keys plus type/len/shape of each value) into the prompt and instructing dynamic key iteration — the first attempt produced 8–9 valid figures for the same runs, with no other change.
• Downstream effect is severe: with empty figures/, the writeup stage produces a paper without result figures (or hallucinates around them).

Repro sketch

1. Complete a BFTS run whose stages produced experiment_data.npy files with several top-level keys (any multi-dataset or ablation-heavy idea does this naturally).
2. Run aggregate_plots as the launcher does.
3. Inspect the generated auto_plot_aggregator.py: it typically indexes one or two literal key names. Check figures/: empty or blank axes, while the log claims success.
4. Manually prepend a structure manifest of the .npy files to the prompt and re-run: figures appear on the first attempt.

Suggested fix (minimal)

In aggregate_plots, before building the prompt, walk every path collected from the summaries' exp_results_npy_files, load each file (np.load(path, allow_pickle=True).item() — these are the framework's own run artifacts, and the generated experiment/plotting code already loads them with allow_pickle=True, so this introduces no new trust boundary), and emit a compact manifest, e.g. per file: list of top-level keys and, for each value, its Python type / length / array shape (optionally one example leaf). Append this manifest to build_aggregator_prompt and add one instruction: iterate over keys discovered at runtime, never hardcode key names, and treat a figure with no plotted data as a failure.

Two cheap complements:

• include the same manifest in the reflection prompt so reflections can correct key handling;
• gate the "ran successfully" log on figure_count > 0 (the count is already computed for the reflection prompt).