[Bug]: ScoreRetryLoop treats zero API calls as infinity when breaking score ties
Describe the bug
ScoreRetryLoop.get_best is documented in code to choose the shortest submission when multiple submissions have the same review score.
The tie-break key uses model_stats.api_calls as the length proxy:
max_indices = sorted(
max_indices,
key=lambda i: self._submissions[i].model_stats.api_calls or float("inf"),
)This treats api_calls=0 the same as a missing value, because 0 is falsy in Python.
As a result, a tied submission with api_calls=0 is sorted after a tied submission with a positive call count. In the reproduction below, get_best() returns index 1, even though index 0 has fewer API calls.
Steps/commands/code to Reproduce
Check out SWE-agent
mainat commit3ea751c087f32b16e039a2233dd6eefecef325d5.Install SWE-agent from source using the repository's development setup.
Create
tests/test_score_retry_loop_zero_api_calls.py:
from types import SimpleNamespace
from sweagent.agent.models import InstanceStats
from sweagent.agent.reviewer import (
ReviewSubmission,
ReviewerResult,
ScoreRetryLoop,
)
def test_score_retry_loop_prefers_zero_api_calls_for_tied_scores():
loop = object.__new__(ScoreRetryLoop)
loop.logger = SimpleNamespace(
debug=lambda *args, **kwargs: None,
info=lambda *args, **kwargs: None,
)
loop._reviews = [
ReviewerResult(
accept=0.9,
outputs=[],
messages=[],
),
ReviewerResult(
accept=0.9,
outputs=[],
messages=[],
),
]
loop._submissions = [
ReviewSubmission(
trajectory=[],
info={},
model_stats=InstanceStats(api_calls=0),
),
ReviewSubmission(
trajectory=[],
info={},
model_stats=InstanceStats(api_calls=5),
),
]
assert loop.get_best() == 0- Run:
python -m pytest tests/test_score_retry_loop_zero_api_calls.py -q- Observe that the assertion fails because
get_best()returns1.
Error message/results
AssertionError: assert 1 == 0Observed behavior:
loop._submissions[0].model_stats.api_calls
# 0
loop._submissions[1].model_stats.api_calls
# 5
loop.get_best()
# 1Expected behavior:
loop.get_best()
# 0Both submissions have the same review score. Since the implementation comment says tied scores should choose the shortest submission, the zero-call submission should be selected before the five-call submission.
The current key evaluates as:
0 or float("inf")
# inf
5 or float("inf")
# 5so the positive call count incorrectly sorts first.
System Information
macOS arm64
Python 3.11+
SWE-agent 1.1.0, main@3ea751c087f32b16e039a2233dd6eefecef325d5Checklist
- I'm running with the latest docker container/on the latest development version (i.e., I ran
git pull)) - I have copied the full command/code that I ran (as text, not as screenshot!)
- If applicable: I have copied the full log file/error message that was the result (as text, not as screenshot!)
- I have enclosed code/log messages in triple backticks (docs) and clicked "Preview" to make sure it's displayed correctly.
Source: SWE-agent/SWE-agent