Bug: Evaluators crash when return_checking is None while appending diagnostics

Author: yifanxiong272Created Aug 29, 2026Updated Sep 9, 2026

Labelsbug

Summary

CoSTEERSingleFeedback.return_checking is declared as str | None, and other code paths already handle None as a valid value.

However, the workflow and pipeline evaluators append validation diagnostics with +=:

python

wfb.return_checking += "\n" + score_check_text

If the feedback object has return_checking=None and a deterministic validation check fails, the evaluator raises a TypeError instead of returning structured negative feedback.

To Reproduce

Check out RD-Agent main at commit 6762f84f9bc0f5c6486c50a00e128a57ac6c3683.
Install RD-Agent from source.
Create test/scenarios/data_science/test_return_checking_none_diagnostics.py:

python

from pathlib import Path
from types import SimpleNamespace

import rdagent.components.coder.data_science.workflow.eval as workflow_eval
from rdagent.components.coder.CoSTEER.evaluators import CoSTEERSingleFeedback
from rdagent.components.coder.data_science.workflow.eval import (
    WorkflowGeneralCaseSpecEvaluator,
)


class FakeTemplate:
    def r(self, *args, **kwargs):
        return "template"


class FakeTask:
    def get_task_information(self):
        return "demo task"


class FakeImplementation:
    def __init__(self, workspace_path):
        self.workspace_path = Path(workspace_path)
        self.file_dict = {
            "main.py": "pass",
            "spec/workflow.md": "spec",
        }

    def execute(self, env=None, entry=None):
        return "inference finished"

    def inject_files(self, **files):
        self.file_dict.update(files)

    def run(self, env=None, entry=None):
        return SimpleNamespace(stdout="submission check passed", exit_code=0)


def test_workflow_evaluator_handles_none_return_checking(tmp_path, monkeypatch):
    monkeypatch.setattr(workflow_eval, "T", lambda *args, **kwargs: FakeTemplate())
    monkeypatch.setattr(
        workflow_eval,
        "get_ds_env",
        lambda *args, **kwargs: SimpleNamespace(
            conf=SimpleNamespace(running_timeout_period=1),
        ),
    )
    monkeypatch.setattr(
        workflow_eval,
        "get_clear_ws_cmd",
        lambda *args, **kwargs: "true",
    )
    monkeypatch.setattr(
        workflow_eval,
        "build_cls_from_json_with_retry",
        lambda *args, **kwargs: CoSTEERSingleFeedback(
            execution="ok",
            return_checking=None,
            code="ok",
            final_decision=True,
        ),
    )

    (tmp_path / "scores.csv").write_text(
        "model,wrong_metric\nensemble,0.5\n",
        encoding="utf-8",
    )

    scen = SimpleNamespace(
        debug_path="/tmp/demo-input",
        real_debug_timeout=lambda: 1,
        metric_name="rmse",
        get_scenario_all_desc=lambda eda_output=None: "scenario",
    )
    evaluator = WorkflowGeneralCaseSpecEvaluator(scen=scen)

    feedback = evaluator.evaluate(
        FakeTask(),
        FakeImplementation(tmp_path),
        None,
    )

    assert feedback.final_decision is False
    assert feedback.return_checking is not None
    assert "column names" in feedback.return_checking

Run:

bash

python -m pytest \
    test/scenarios/data_science/test_return_checking_none_diagnostics.py \
    -q

Observe that the test fails with an uncaught TypeError.

Expected Behavior

A schema-valid feedback object with return_checking=None should not crash the evaluator.

When a deterministic validation check fails, the evaluator should normalize None to an empty string, set:

python

feedback.final_decision is False

and append a readable diagnostic to feedback.return_checking.

Actual Behavior

The evaluator raises:

TypeError: unsupported operand type(s) for +=: 'NoneType' and 'str'

No feedback object is returned.

Screenshot

Not applicable; this is a deterministic unit-level reproduction.

Environment

Name of current operating system: macOS
Processor architecture: arm64
Python version: 3.11.15
RD-Agent version: 0.8.0, main@6762f84f9bc0f5c6486c50a00e128a57ac6c3683
Container: not used in this reproduction

Additional Notes

CoSTEERSingleFeedback declares return_checking as nullable:

python

return_checking: str | None

The workflow evaluator then appends diagnostics without normalizing it:

python

if score_ret_code != 0:
    wfb.final_decision = False
    wfb.return_checking += "\n" + score_check_text

The pipeline evaluator has the same pattern:

python

if score_ret_code != 0 and wfb.final_decision is True:
    wfb.final_decision = False
    wfb.return_checking += "\n" + score_check_text

By contrast, ModelDumpEvaluator already uses a None-safe append pattern:

python

csfb.return_checking = (csfb.return_checking or "") + return_msg

A possible fix is to use the same normalization before appending diagnostics in the workflow and pipeline evaluators.

Source: microsoft/RD-Agent

View original on GitHub View discussion on GitHub