Tree search: a single failed worker node aborts the entire run (`raise` in result-collection loop of `ParallelAgent.step`)

Related issues: RELATED #65 (closed) — that report shows the same failure class (a worker's JSONDecodeError propagating through future.result() and killing the whole run), but only as a symptom/question. This report identifies the explicit catch-and-re-raise in the result-collection loop as the root cause and proposes a fix. No open duplicate found.

Summary

In ParallelAgent.step, the loop that collects worker results catches any exception per future, logs it — and then re-raises it. The re-raised exception propagates out of manager.run() and terminates the entire experiment, so one bad node (malformed LLM completion, truncated/empty API response, a transient backend error, or buggy generated code surfacing in the worker) kills a tree search that may have dozens of healthy nodes and hours of accumulated compute.

Tree search is explicitly designed to tolerate failed nodes (buggy nodes are part of the search), so a single worker failure aborting the whole run defeats the design.

Where

ai_scientist/treesearch/parallel_agent.py, lines 2148–2181 at current main (96bd516), inside ParallelAgent.step:

        # Add results to journal
        print("Waiting for results")
        for i, future in enumerate(futures):
            try:
                print("About to get result from future")
                result_data = future.result(timeout=self.timeout)
                ...
                # Add node to journal's list and assign its step number
                self.journal.append(result_node)
                print("Added result node to journal")

            except TimeoutError:
                print("Worker process timed out, couldn't get the result")
                logger.error(f"Worker process timed out, couldn't get the result")
            except Exception as e:
                print(f"Error processing node: {str(e)}")
                logger.error(f"Error processing node: {str(e)}")
                import traceback

                traceback.print_exc()
                raise          # <-- single node failure kills the whole run
            finally:
                ...

Note the asymmetry: a TimeoutError is logged and tolerated, but any other exception — including ones originating inside the worker and merely re-delivered by future.result() — is fatal.

Impact

• Measured on a 5-run full-pipeline benchmark (same ideas, several LLM backends): 2/5 runs completed end-to-end with the raise in place; 4/5 completed after changing it to log-and-skip. The aborted runs died non-deterministically mid-stage on a single node's exception.
• Failures typically occur hours into a run, wasting all prior compute, and (because the abort path skips later pipeline steps) leaving no writeup/review.
• Closed issue #65 shows a user hitting the same class of failure in the wild (JSONDecodeError from an empty model response propagating through future.result()).

Repro sketch

1. Launch launch_scientist_bfts.py with any backend that occasionally returns malformed or empty completions (most non-OpenAI gateways, or OpenAI under heavy rate limiting).
2. Alternatively, deterministically: inject raise RuntimeError("boom") into one worker's result path (e.g. in Node.from_dict for a specific node id).
3. Observe the entire run abort with a traceback through manager.run() even though all other parallel workers returned healthy results.

Suggested fix (minimal)

In the result-collection loop, guard only the retrieval and skip the bad node:

• Wrap just result_data = future.result(timeout=self.timeout) in its own try/except; on TimeoutError or a generic worker exception, log + continue to the next future instead of raise.
• Keep one fatal case: concurrent.futures.process.BrokenProcessPool should still be re-raised — when the pool is broken every remaining future will fail too, and silently skipping all of them would corrupt the stage.
• Leave the journal-integration code (Node.from_dict, state updates, journal.append) outside the tolerant guard so genuine framework/state bugs still surface instead of silently corrupting the tree.

This is a small, contained diff (replace the trailing raise with continue, plus the BrokenProcessPool carve-out) and in our testing doubled end-to-end success rate.