Bug: analyze_error returns a matched error as both node and raw string
Bug Description
CoSTEERRAGStrategyV2.analyze_error can return two entries for the same parsed error content when the knowledge graph contains both:
- one existing error node that matches the current error content; and
- at least one other error node that does not match it.
The method comment describes the output as:
# Raised errors, existed error nodes + not existed error nodes(here, they are strs)So a parsed error content that already exists in the graph should be represented by the existing error node, while a parsed error content that does not exist should remain a string.
However, the current implementation loops over every existing error node and appends either the matching node or the raw string for each graph node. For one matched content, this can produce both:
"same error content"and:
UndirectedNode(content="same error content", label="error")in the returned list.
To Reproduce
Steps to reproduce the behavior:
- Check out RD-Agent
mainat commit:
2c878f9d2453dced35061165786d1f31bbff0ab6- Install RD-Agent from source and install pytest:
python -m pip install -e .
python -m pip install pytest- Create
test/test_costeer_analyze_error_duplicate_match.py:
from types import SimpleNamespace
from rdagent.components.coder.CoSTEER.knowledge_management import (
CoSTEERRAGStrategyV2,
)
from rdagent.components.knowledge_management.graph import (
UndirectedNode,
)
class GraphStub:
def __init__(self, nodes):
self.nodes = nodes
def get_all_nodes_by_label_list(self, labels):
assert labels == ["error"]
return self.nodes
def test_analyze_error_returns_matched_node_once():
matched_content = (
"The source dataframe and the ground truth dataframe "
"have different rows count."
)
missing_content = (
"Some values differ by more than the tolerance of 1e-6."
)
unrelated_node = UndirectedNode(
content="A different previous error.",
label="error",
)
matched_node = UndirectedNode(
content=matched_content,
label="error",
)
strategy = CoSTEERRAGStrategyV2.__new__(
CoSTEERRAGStrategyV2
)
strategy.knowledgebase = SimpleNamespace(
graph=GraphStub(
[
unrelated_node,
matched_node,
]
)
)
feedback = (
matched_content
+ "\n"
+ missing_content
)
result = strategy.analyze_error(
feedback,
feedback_type="value",
)
assert result == [
matched_node,
missing_content,
]- Run:
python -m pytest \
test/test_costeer_analyze_error_duplicate_match.py \
-q- Observe that the test fails because the returned list contains three items instead of two.
Representative observed result:
[
"The source dataframe and the ground truth dataframe have different rows count.",
UndirectedNode(
content=(
"The source dataframe and the ground truth dataframe "
"have different rows count."
),
label="error",
),
"Some values differ by more than the tolerance of 1e-6.",
]The first two entries represent the same parsed error content.
Expected Behavior
analyze_error should return exactly one output item for each distinct parsed error content.
For a parsed error that already has a matching graph node, the returned item should be that existing node:
matched_nodeFor a parsed error that does not have a matching graph node, the returned item should remain the original string:
missing_contentFor the reproduction above, the result should be:
[
matched_node,
missing_content,
]Screenshot
Not applicable. This is a deterministic unit-level reproduction.
Environment
Note: Users can run rdagent collect_info to get system information and paste it
directly here.
- Name of current operating system: macOS
- Processor architecture: arm64
- System, version, and hardware information: macOS 15.7.3, arm64
- Version number of the system: 15.7.3
- Python version: 3.13.2
- Container ID: Not applicable
- Container Name: Not applicable
- Container Status: Not applicable
- Image ID used by the container: Not applicable
- Image tag used by the container: Not applicable
- Container port mapping: Not applicable
- Container Label: Not applicable
- Startup Commands: Not applicable
- RD-Agent version:
main@2c878f9d2453dced35061165786d1f31bbff0ab6 - Package version: Source checkout
Additional Notes
The current implementation parses error_contents, retrieves all graph error nodes, and then loops over both collections:
all_error_nodes = (
self.knowledgebase.graph
.get_all_nodes_by_label_list(["error"])
)
if not len(all_error_nodes):
return error_contents
else:
error_list = []
for error_content in error_contents:
for error_node in all_error_nodes:
if error_content == error_node.content:
error_list.append(error_node)
else:
error_list.append(error_content)
if error_list[-1] in error_list[:-1]:
error_list.pop()
return error_listThe problem is that the else branch runs once for every nonmatching graph node. If an unrelated error node is checked before or after the matching node, the raw string is appended in addition to the matched node.
The ad-hoc duplicate removal does not remove this pair because the string and UndirectedNode object are not equal.
This result is later saved into working trace error analysis:
self.knowledgebase.working_trace_error_analysis.setdefault(
target_task_information,
[],
).append(error_analysis_result)and reused by error_query, which turns non-node entries back into graph nodes by content before querying similar successful knowledge.
A possible fix is to decide once per parsed error_content: find the first matching node, append that node if found, otherwise append the string.
For example:
error_list = []
for error_content in error_contents:
matched_node = next(
(
error_node
for error_node in all_error_nodes
if error_node.content == error_content
),
None,
)
error_list.append(
matched_node
if matched_node is not None
else error_content
)Regression coverage should include:
- no existing error nodes;
- one parsed error with a matching existing node;
- one parsed error with only nonmatching existing nodes;
- multiple parsed value-check errors with one match and one nonmatch;
- graph nodes ordered with the unrelated node before the matching node;
- graph nodes ordered with the matching node before the unrelated node;
- preservation of parsed error-content order;
- no duplicate string/node pair for the same error content.
Targeted issue and pull-request searches found no existing report for this analyze_error matched-node duplication root.
Source: microsoft/RD-Agent