#1527·SWE-agent

[Bug]: ScoreRetryLoop treats zero API calls as infinity when breaking score ties

Author: yifanxiong272Created Aug 31, 2026Updated Aug 31, 2026

Describe the bug

ScoreRetryLoop.get_best is documented in code to choose the shortest submission when multiple submissions have the same review score.

The tie-break key uses model_stats.api_calls as the length proxy:

python
max_indices = sorted(
    max_indices,
    key=lambda i: self._submissions[i].model_stats.api_calls or float("inf"),
)

This treats api_calls=0 the same as a missing value, because 0 is falsy in Python.

As a result, a tied submission with api_calls=0 is sorted after a tied submission with a positive call count. In the reproduction below, get_best() returns index 1, even though index 0 has fewer API calls.

Steps/commands/code to Reproduce

  1. Check out SWE-agent main at commit 3ea751c087f32b16e039a2233dd6eefecef325d5.

  2. Install SWE-agent from source using the repository's development setup.

  3. Create tests/test_score_retry_loop_zero_api_calls.py:

python
from types import SimpleNamespace

from sweagent.agent.models import InstanceStats
from sweagent.agent.reviewer import (
    ReviewSubmission,
    ReviewerResult,
    ScoreRetryLoop,
)


def test_score_retry_loop_prefers_zero_api_calls_for_tied_scores():
    loop = object.__new__(ScoreRetryLoop)
    loop.logger = SimpleNamespace(
        debug=lambda *args, **kwargs: None,
        info=lambda *args, **kwargs: None,
    )

    loop._reviews = [
        ReviewerResult(
            accept=0.9,
            outputs=[],
            messages=[],
        ),
        ReviewerResult(
            accept=0.9,
            outputs=[],
            messages=[],
        ),
    ]
    loop._submissions = [
        ReviewSubmission(
            trajectory=[],
            info={},
            model_stats=InstanceStats(api_calls=0),
        ),
        ReviewSubmission(
            trajectory=[],
            info={},
            model_stats=InstanceStats(api_calls=5),
        ),
    ]

    assert loop.get_best() == 0
  1. Run:
bash
python -m pytest tests/test_score_retry_loop_zero_api_calls.py -q
  1. Observe that the assertion fails because get_best() returns 1.

Error message/results

AssertionError: assert 1 == 0

Observed behavior:

python
loop._submissions[0].model_stats.api_calls
# 0

loop._submissions[1].model_stats.api_calls
# 5

loop.get_best()
# 1

Expected behavior:

python
loop.get_best()
# 0

Both submissions have the same review score. Since the implementation comment says tied scores should choose the shortest submission, the zero-call submission should be selected before the five-call submission.

The current key evaluates as:

python
0 or float("inf")
# inf

5 or float("inf")
# 5

so the positive call count incorrectly sorts first.

System Information

macOS arm64
Python 3.11+
SWE-agent 1.1.0, main@3ea751c087f32b16e039a2233dd6eefecef325d5

Checklist

  • I'm running with the latest docker container/on the latest development version (i.e., I ran git pull))
  • I have copied the full command/code that I ran (as text, not as screenshot!)
  • If applicable: I have copied the full log file/error message that was the result (as text, not as screenshot!)
  • I have enclosed code/log messages in triple backticks (docs) and clicked "Preview" to make sure it's displayed correctly.