Bug: DSCoSTEERRunner treats tied minimization scores as improvements

Summary

DSCoSTEERRunner.should_use_new_evo treats tied scores inconsistently between maximization and minimization objectives.

When self.scen.metric_direction is True (higher is better), a tied score is not selected:

new.score == base.score
# should_use_new_evo(...) returns False

When self.scen.metric_direction is False (lower is better), the same tied score is selected:

new.score == base.score
# should_use_new_evo(...) returns True

This appears to come from a boolean comparison that conflates "not greater than" with "strictly lower than" for minimization. The current expression effectively makes minimization use "not greater than" instead of "less than".

To Reproduce

1. Check out RD-Agent main at commit 6762f84f9bc0f5c6486c50a00e128a57ac6c3683.

2. Install RD-Agent from source.

3. Create test/scenarios/data_science/test_runner_tie_score_policy.py:

from types import SimpleNamespace

from rdagent.components.coder.CoSTEER.evaluators import CoSTEERMultiFeedback
from rdagent.scenarios.data_science.dev.runner import DSCoSTEERRunner
from rdagent.scenarios.data_science.dev.runner.eval import DSRunnerFeedback


def make_feedback(score):
    return CoSTEERMultiFeedback([
        DSRunnerFeedback(
            execution="ok",
            return_checking="ok",
            code="ok",
            final_decision=True,
            acceptable=True,
            score=score,
        )
    ])


def make_runner(metric_direction):
    runner = object.__new__(DSCoSTEERRunner)
    runner.scen = SimpleNamespace(metric_direction=metric_direction)
    return runner


def test_tied_scores_are_not_treated_as_improvements_for_minimization():
    base = make_feedback(1.0)
    tied = make_feedback(1.0)

    assert make_runner(metric_direction=True).should_use_new_evo(
        base,
        tied,
    ) is False
    assert make_runner(metric_direction=False).should_use_new_evo(
        base,
        tied,
    ) is False

1. Run:

python -m pytest test/scenarios/data_science/test_runner_tie_score_policy.py -q

1. Observe that the minimization assertion fails.

Expected Behavior

A tied score should not be treated as a strict improvement.

For consistency with the current maximization behavior:

base.score == new.score

should return False for both higher-is-better and lower-is-better metrics.

If accepting ties is intended, the policy should be explicit and symmetric across both metric directions.

Actual Behavior

The tied maximization case returns False, but the tied minimization case returns True:

AssertionError: assert True is False

So a lower-is-better run can replace the current candidate even though the new candidate did not improve the score.

Screenshot

Not applicable; this is a deterministic unit-level reproduction.

Environment

• Name of current operating system: macOS
• Processor architecture: arm64
• Python version: 3.11.15
• RD-Agent version: 0.8.0, main@6762f84f9bc0f5c6486c50a00e128a57ac6c3683
• Container: not used in this reproduction

Additional Notes

The current implementation is:

def compare_scores(s1, s2) -> bool:
    if s2 is None:
        return False
    if s1 is None:
        return True
    return (s2 > s1) == self.scen.metric_direction

For a minimization objective, self.scen.metric_direction is False.

When the scores are tied:

s2 > s1
# False

Therefore the expression becomes:

False == False
# True

This makes equality count as an improvement only for minimization objectives.

A clearer implementation would use explicit directional comparisons:

if self.scen.metric_direction:
    return s2 > s1
return s2 < s1

Alternatively, if ties should be accepted, both directions should use an explicit and documented tie policy such as >= / <=.

This matters because should_use_new_evo decides whether a newly generated candidate replaces the current one. In lower-is-better data-science runs, accepting a tied candidate can cause unnecessary candidate churn or replace a stable baseline with a version that has no metric improvement.