Bug: ModelDumpEvaluator crashes on an empty scores.csv instead of returning failure feedback

Author: yifanxiong272Created Aug 29, 2026Updated Aug 31, 2026

Labelsbug

Summary

ModelDumpEvaluator.evaluate raises an uncaught pandas.errors.EmptyDataError when scores.csv exists but is empty.

The evaluator already treats missing scores.csv as a structured failure and returns a CoSTEERSingleFeedback with final_decision=False. An empty scores.csv should be handled the same way: it is invalid model output, but it should not crash the evaluator.

To Reproduce

Check out RD-Agent main at commit 6762f84f9bc0f5c6486c50a00e128a57ac6c3683.
Install RD-Agent from source.
Create test/qlib/test_model_dump_empty_scores.py:

python

from types import SimpleNamespace

import rdagent.components.coder.data_science.share.eval as share_eval
from rdagent.components.coder.data_science.share.eval import ModelDumpEvaluator


class FakeImplementation:
    all_codes = {"main.py": "pass"}

    def __init__(self, workspace_path):
        self.workspace_path = workspace_path

    def execute(self, env=None, entry=None):
        return "inference finished"


def test_model_dump_evaluator_handles_empty_scores_csv(tmp_path, monkeypatch):
    monkeypatch.setattr(share_eval, "get_ds_env", lambda *args, **kwargs: None)
    monkeypatch.setattr(
        share_eval,
        "get_clear_ws_cmd",
        lambda *args, **kwargs: "true",
    )

    (tmp_path / "models").mkdir()
    (tmp_path / "models" / "model.bin").write_text("model", encoding="utf-8")
    (tmp_path / "submission.csv").write_text(
        "id,pred\n1,0.5\n",
        encoding="utf-8",
    )
    (tmp_path / "scores.csv").write_text("", encoding="utf-8")

    scen = SimpleNamespace(
        competition="demo-competition",
        debug_path="/tmp/demo-input",
        real_debug_timeout=lambda: 1,
        real_full_timeout=lambda: 1,
    )

    evaluator = ModelDumpEvaluator(scen, data_type="sample")
    feedback = evaluator.evaluate(
        None,
        FakeImplementation(tmp_path),
        None,
    )

    assert feedback.final_decision is False
    assert "scores.csv" in feedback.return_checking

Run:

bash

python -m pytest test/qlib/test_model_dump_empty_scores.py -q

Observe that the test fails with an uncaught EmptyDataError.

Expected Behavior

An empty scores.csv should be treated as invalid generated output. evaluate should return structured negative feedback, for example:

python

feedback.final_decision is False

and the feedback should mention that scores.csv is empty or cannot be parsed.

Actual Behavior

evaluate raises an uncaught pandas exception:

pandas.errors.EmptyDataError: No columns to parse from file

As a result, the evaluator crashes instead of returning CoSTEERSingleFeedback.

Screenshot

Not applicable; this is a deterministic unit-level reproduction.

Environment

Name of current operating system: macOS
Processor architecture: arm64
Python version: 3.11.15
RD-Agent version: 0.8.0, main@6762f84f9bc0f5c6486c50a00e128a57ac6c3683
Package version: pandas 2.3.3, pytest 9.1.1
Container: not used in this reproduction

Additional Notes

The crash occurs here:

python

score_df = pd.read_csv(
    (implementation.workspace_path / "scores.csv"),
    index_col=0,
)

There is an existence check immediately before this block, but no parse-error handling for the present-but-empty file case.

A possible fix is to catch pd.errors.EmptyDataError and other CSV parse errors around this read, then return CoSTEERSingleFeedback(final_decision=False, ...) with a clear diagnostic.

Source: microsoft/RD-Agent

View original on GitHub View discussion on GitHub