Bug: ModelDumpEvaluator drops changed-artifact diagnostics in high check mode

Author: yifanxiong272Created Aug 29, 2026Updated Aug 29, 2026

Labelsbug

Summary

ModelDumpEvaluator.evaluate does not independently report changes to scores.csv and submission.csv when DS_RD_SETTING.model_dump_check_level is set to "high".

Two cases are affected:

If only submission.csv changes, no diagnostic is added.
If both files change, the submission.csv message overwrites the previously constructed scores.csv message, including its before-and-after contents.

Consequently, the high-level deterministic check can return incomplete or empty diagnostics even though one or both monitored artifacts changed during inference.

To Reproduce

Check out RD-Agent main at commit 6762f84f9bc0f5c6486c50a00e128a57ac6c3683.
Install RD-Agent from source.
Create test/scenarios/data_science/test_model_dump_high_diagnostics.py:

python

from types import SimpleNamespace

import pytest

import rdagent.components.coder.data_science.share.eval as share_eval
from rdagent.components.coder.CoSTEER.evaluators import (
    CoSTEERSingleFeedback,
)
from rdagent.components.coder.data_science.share.eval import (
    ModelDumpEvaluator,
)


class FakeTemplate:
    def r(self, *args, **kwargs):
        return "/input"


@pytest.mark.parametrize(
    ("change_scores", "change_submission", "expected_files"),
    [
        (False, True, ("submission.csv",)),
        (True, True, ("scores.csv", "submission.csv")),
    ],
)
def test_model_dump_high_check_reports_every_changed_artifact(
    tmp_path,
    monkeypatch,
    change_scores,
    change_submission,
    expected_files,
):
    monkeypatch.setattr(
        share_eval,
        "T",
        lambda *args, **kwargs: FakeTemplate(),
    )
    monkeypatch.setattr(
        share_eval,
        "get_ds_env",
        lambda *args, **kwargs: object(),
    )
    monkeypatch.setattr(
        share_eval,
        "get_clear_ws_cmd",
        lambda *args, **kwargs: "clear",
    )
    monkeypatch.setattr(
        share_eval,
        "remove_eda_part",
        lambda stdout: stdout,
    )
    monkeypatch.setattr(
        share_eval.DS_RD_SETTING,
        "model_dump_check_level",
        "high",
    )
    monkeypatch.setattr(
        share_eval,
        "build_cls_from_json_with_retry",
        lambda *args, **kwargs: CoSTEERSingleFeedback(
            execution="ok",
            return_checking="",
            code="ok",
            final_decision=True,
        ),
    )

    (tmp_path / "models").mkdir()
    (tmp_path / "models" / "model.bin").write_text(
        "model",
        encoding="utf-8",
    )

    scores_before = "model,score\nensemble,0.1\n"
    scores_after = "model,score\nensemble,0.2\n"
    submission_before = "id,pred\n1,0.1\n"
    submission_after = "id,pred\n1,0.2\n"

    (tmp_path / "scores.csv").write_text(
        scores_before,
        encoding="utf-8",
    )
    (tmp_path / "submission.csv").write_text(
        submission_before,
        encoding="utf-8",
    )

    class FakeImplementation:
        workspace_path = tmp_path
        all_codes = {"main.py": "pass"}

        def execute(self, env=None, entry=None):
            if entry == "clear":
                (tmp_path / "scores.csv").unlink()
                (tmp_path / "submission.csv").unlink()
                return "cleared"

            assert "--inference" in entry

            (tmp_path / "scores.csv").write_text(
                scores_after if change_scores else scores_before,
                encoding="utf-8",
            )
            (tmp_path / "submission.csv").write_text(
                submission_after
                if change_submission
                else submission_before,
                encoding="utf-8",
            )
            return "inference finished"

    scen = SimpleNamespace(
        competition="demo-competition",
        debug_path="/tmp/demo-input",
        real_debug_timeout=lambda: 1,
        real_full_timeout=lambda: 1,
    )
    evaluator = ModelDumpEvaluator(
        scen,
        data_type="sample",
    )

    feedback = evaluator.evaluate(
        None,
        FakeImplementation(),
        None,
    )

    diagnostics = feedback.return_checking or ""

    for filename in expected_files:
        assert (
            f"content of {filename} has changed"
            in diagnostics
        )

Run:

bash

python -m pytest \
    test/scenarios/data_science/test_model_dump_high_diagnostics.py \
    -q \
    --maxfail=0

Observe that both parameterized cases fail.

Expected Behavior

When high-level checking is enabled, each changed artifact should be reported independently.

For a submission-only change, return_checking should mention:

The content of submission.csv has changed

When both files change, it should contain both diagnostics:

The content of scores.csv has changed
The content of submission.csv has changed

The detailed before-and-after information for scores.csv should also be preserved.

Actual Behavior

When only submission.csv changes:

python

feedback.return_checking
# ""

When both files change, only the submission message remains:

[Error] The content of submission.csv has changed. ...

The scores.csv message and its before-and-after contents are lost.

Representative failures:

AssertionError: assert 'content of submission.csv has changed' in ''

AssertionError: assert 'content of scores.csv has changed' in '[Error] The content of submission.csv has changed. ...'

Impact

High check mode is intended to detect whether inference unexpectedly regenerates monitored artifacts instead of consistently reusing the dumped model.

A submission-only change is currently invisible to the deterministic diagnostic path. When both files change, evidence about the changed scores is discarded.

Callers consequently receive incomplete information about inconsistent inference output, making model-dump failures harder to identify and debug. Downstream callers are left with the separately generated feedback and incomplete deterministic artifact-change evidence.

Screenshot

Not applicable; this is a deterministic unit-level reproduction.

Environment

Name of current operating system: macOS
Processor architecture: arm64
Python version: 3.11.15
RD-Agent version: 0.8.0, main@6762f84f9bc0f5c6486c50a00e128a57ac6c3683
Package version: pandas 2.3.3, pytest 9.1.1
Container: not used in this reproduction

Additional Notes

The current implementation nests the submission check inside the scores check:

python

if scores_content_before != scores_content_after:
    return_msg = (
        "\n[Error] The content of scores.csv has changed. ..."
    )
    return_msg += (
        f"\nBefore:\n{scores_content_before}"
        f"\nAfter:\n{scores_content_after}"
    )

    if submission_content_before != submission_content_after:
        return_msg = (
            "[Error] The content of submission.csv "
            "has changed. ..."
        )

    csfb.return_checking = (
        csfb.return_checking or ""
    ) + return_msg

This causes two separate control-flow problems:

the nested submission condition is never evaluated when only submission.csv changes;
the assignment in the nested condition replaces the existing scores diagnostic when both files change.

A possible fix is to construct the diagnostics independently and append every applicable message:

python

messages = []

if scores_content_before != scores_content_after:
    messages.append(scores_change_message)

if submission_content_before != submission_content_after:
    messages.append(submission_change_message)

if messages:
    csfb.return_checking = (
        csfb.return_checking or ""
    ) + "\n".join(messages)

Regression coverage should include:

only scores.csv changing;
only submission.csv changing;
both files changing;
neither file changing;
preservation of before-and-after contents;
preservation of an existing return_checking value.

Source: microsoft/RD-Agent

View original on GitHub View discussion on GitHub