Bug: scores.csv error handling can crash on undecodable file content

Author: yifanxiong272Created Aug 29, 2026Updated Aug 29, 2026

Labelsbug

Summary

Several data-science evaluators catch errors while parsing scores.csv, but then read the same file with Path.read_text() while constructing the diagnostic message.

If scores.csv exists but contains bytes that are not valid UTF-8, the initial pandas.read_csv(...) failure is caught, but the exception handler raises a second uncaught UnicodeDecodeError:

python

score_fp.read_text()

As a result, the evaluator crashes instead of returning structured negative feedback.

To Reproduce

Check out RD-Agent main at commit 6762f84f9bc0f5c6486c50a00e128a57ac6c3683.
Install RD-Agent from source.
Create test/scenarios/data_science/test_scores_csv_decode_error.py:

python

from pathlib import Path
from types import SimpleNamespace

import rdagent.components.coder.data_science.workflow.eval as workflow_eval
from rdagent.components.coder.CoSTEER.evaluators import CoSTEERSingleFeedback
from rdagent.components.coder.data_science.workflow.eval import (
    WorkflowGeneralCaseSpecEvaluator,
)


class FakeTask:
    def get_task_information(self):
        return "demo task"


class FakeImplementation:
    def __init__(self, workspace_path):
        self.workspace_path = Path(workspace_path)
        self.file_dict = {
            "main.py": "pass",
            "spec/workflow.md": "spec",
        }

    def execute(self, env=None, entry=None):
        return "inference finished"

    def inject_files(self, **files):
        self.file_dict.update(files)

    def run(self, env=None, entry=None):
        return SimpleNamespace(stdout="submission check passed", exit_code=0)


def test_workflow_evaluator_handles_undecodable_scores_csv(tmp_path, monkeypatch):
    monkeypatch.setattr(
        workflow_eval,
        "get_ds_env",
        lambda *args, **kwargs: SimpleNamespace(
            conf=SimpleNamespace(running_timeout_period=1),
        ),
    )
    monkeypatch.setattr(
        workflow_eval,
        "get_clear_ws_cmd",
        lambda *args, **kwargs: "true",
    )
    monkeypatch.setattr(
        workflow_eval,
        "build_cls_from_json_with_retry",
        lambda *args, **kwargs: CoSTEERSingleFeedback(
            execution="ok",
            return_checking="ok",
            code="ok",
            final_decision=True,
        ),
    )

    (tmp_path / "scores.csv").write_bytes(b"\xff\xff\xff\n")

    scen = SimpleNamespace(
        debug_path="/tmp/demo-input",
        real_debug_timeout=lambda: 1,
        get_scenario_all_desc=lambda eda_output=None: "scenario",
    )
    evaluator = WorkflowGeneralCaseSpecEvaluator(scen=scen)

    feedback = evaluator.evaluate(
        FakeTask(),
        FakeImplementation(tmp_path),
        None,
    )

    assert feedback.final_decision is False
    assert "scores.csv" in feedback.return_checking

Run:

bash

python -m pytest test/scenarios/data_science/test_scores_csv_decode_error.py -q

Observe that the test fails with an uncaught UnicodeDecodeError.

Expected Behavior

An undecodable or malformed scores.csv should be treated as invalid generated output.

The evaluator should return structured negative feedback, for example:

python

feedback.final_decision is False

and include a diagnostic mentioning the scores.csv parse/decode problem.

Actual Behavior

The evaluator raises an uncaught exception:

UnicodeDecodeError: 'utf-8' codec can't decode byte 0xff in position 0: invalid start byte

So no feedback object is returned.

Screenshot

Not applicable; this is a deterministic unit-level reproduction.

Environment

Name of current operating system: macOS
Processor architecture: arm64
Python version: 3.11.15
RD-Agent version: 0.8.0, main@6762f84f9bc0f5c6486c50a00e128a57ac6c3683
Container: not used in this reproduction

Additional Notes

The workflow evaluator contains this exception handler:

python

except Exception as e:
    score_check_text += f"\n[Error] in checking the scores.csv file: {e}\nscores.csv's content:\n-----\n{score_fp.read_text()}\n-----"
    score_ret_code = 1

The same pattern also appears in:

rdagent/components/coder/data_science/pipeline/eval.py
rdagent/scenarios/data_science/dev/runner/eval.py

The handler catches the first parsing error, but score_fp.read_text() can fail for the same artifact before the evaluator can set score_ret_code and return feedback.

A possible fix is to make diagnostic file reading best-effort, for example by using errors="replace" or catching decode errors separately before appending the raw file content to the feedback.

Source: microsoft/RD-Agent

View original on GitHub View discussion on GitHub