BUG: BinaryTrajectoryComparison skips extra sampling for indented edit actions
Describe the bug
Summary
In sweagent/agent/action_sampler.py, BinaryTrajectoryComparison.contains_edits checks parsed actions with action.startswith(...) without removing leading whitespace.
ThoughtActionParser preserves indentation inside the action code block, while the execution path strips the parsed action before running it. An action such as edit file.txt is therefore normalized to edit file.txt for execution, but contains_edits returns False.
Impact
BinaryTrajectoryComparison.get_completions uses contains_edits to decide whether to request additional completions up to max_n_samples.
When an edit action is indented inside the model's code block, the configured edit-triggered extra-sampling branch is skipped even though the execution path normalizes it to the same edit command.
Steps/commands/code to Reproduce
Check out SWE-agent
mainat commit3ea751c087f32b16e039a2233dd6eefecef325d5.Set up the source checkout:
python3.12 -m venv .venv
source .venv/bin/activate
pip install -e '.[dev]'- Add the following test to
tests/test_parsing.py:
def test_binary_trajectory_comparison_detects_indented_edit():
from sweagent.agent.action_sampler import BinaryTrajectoryComparison
from sweagent.tools.parsing import ThoughtActionParser
from sweagent.tools.tools import ToolConfig, ToolHandler
tools = ToolHandler(ToolConfig(parse_function=ThoughtActionParser()))
sampler = BinaryTrajectoryComparison.__new__(BinaryTrajectoryComparison)
sampler._tools = tools
completion = {
"message": "Plan\n```\n edit file.txt\n```",
}
_, action = tools.parse_actions(completion)
assert tools.guard_multiline_input(action).strip() == "edit file.txt"
assert sampler.contains_edits([completion]) is True- Run:
python -m pytest tests/test_parsing.py -q- Observe that the newly added assertion fails while the eight existing tests pass.
Error message/results
Expected result
ThoughtActionParser returns an action containing the leading indentation. The execution normalization produces:
"edit file.txt"Because this is one of the edit command prefixes recognized by BinaryTrajectoryComparison, contains_edits should return:
TrueActual result
The execution-normalization assertion passes, but contains_edits returns False.
The focused run reports:
........F [100%]
FAILED tests/test_parsing.py::test_binary_trajectory_comparison_detects_indented_edit
AssertionError: assert False is True
+ where False = contains_edits([{'message': 'Plan\n```\n edit file.txt\n```'}])
1 failed, 8 passedAt the tested commit, edit detection operates on the unnormalized action:
def contains_edits(self, completions: list[dict[str, Any]]) -> bool:
keywords = [
"edit",
"str_replace_editor insert",
"str_replace_editor str_replace",
]
for completion in completions:
_, action = self._tools.parse_actions(completion)
if any(action.startswith(keyword) for keyword in keywords):
return True
return FalseThe execution path applies whitespace normalization before sending the same action to the environment:
run_action: str = self.tools.guard_multiline_input(
step.action
).strip()For the focused input, the parser returns " edit file.txt\n". contains_edits does not recognize it, while the execution path produces "edit file.txt".
One possible fix direction is to normalize leading whitespace before testing the edit prefixes:
normalized_action = action.lstrip()
if any(
normalized_action.startswith(keyword)
for keyword in keywords
):
return TrueRegression coverage should include unindented and indented forms of edit, str_replace_editor insert, and str_replace_editor str_replace, together with a non-edit action.
System Information
- SWE-agent:
1.1.0,main@3ea751c087f32b16e039a2233dd6eefecef325d5 - SWE-ReX:
1.4.0 - Python:
3.12.13 - Operating system: macOS
15.7.3,arm64 - Installation: source checkout with editable development installation
- Parser mode:
thought_action - Model/provider: not applicable; reproduced through deterministic parsing and action-sampling code
Checklist
- I'm running with the latest docker container/on the latest development version (i.e., I ran
git pull)) - I have copied the full command/code that I ran (as text, not as screenshot!)
- If applicable: I have copied the full log file/error message that was the result (as text, not as screenshot!)
- I have enclosed code/log messages in triple backticks (docs) and clicked "Preview" to make sure it's displayed correctly.
Source: SWE-agent/SWE-agent