#3010·ragas

AG-UI MessagesSnapshot: convert_messages_snapshot drops tool call name and arguments (ToolCall(name='unknown_tool', args={}))

Author: BlueX888Created Sep 15, 2026Updated Sep 15, 2026
  • I have checked the documentation and related resources and couldn't resolve my bug.

Describe the bug

convert_messages_snapshot() (via AGUIEventCollector._handle_messages_snapshot, src/ragas/integrations/ag_ui.py:636) rebuilds tool calls from a MessagesSnapshotEvent with getattr(tc_obj, "name", "unknown_tool") and getattr(tc_obj, "args", {}). In ag_ui.core, ToolCall has no top-level name / args; both are nested under ToolCall.function (FunctionCall.name: str, FunctionCall.arguments: str — a JSON string). Both getattr calls therefore always fall back, so every tool call recovered from a snapshot comes back as ToolCall(name='unknown_tool', args={}) and the real tool name and arguments are silently discarded.

Root cause: src/ragas/integrations/ag_ui.py:636-637 at commit 298b68274234c060deacab3cf5fb52aa3a20e885 reads the flat shape where the AG-UI object is nested. The sibling writer in the same file, convert_messages_to_ag_ui (lines 925-931), builds the nested form AGUIToolCall(id=..., function=FunctionCall(name=tc.name, arguments=json.dumps(tc.args))), so a Ragas → AG-UI → Ragas round-trip loses every tool name and argument. The streaming path in the same class (_handle_tool_call_end) does parse function.arguments-style JSON, so only the snapshot path is affected.

Ragas version: v0.4.3-8-g298b682 (editable pip install -e . of commit 298b68274234c060deacab3cf5fb52aa3a20e885) Python version: 3.12.14 ag-ui-protocol: 0.1.9 (pyproject.toml pins ag-ui = ["ag-ui-protocol>=0.1.9"]; the nesting was verified in 0.1.9, 0.1.21 and 0.1.22) OS: macOS 26.6.2 (Darwin arm64)

Code to Reproduce

python
# real `pip install -e .` of ragas @ 298b682, no shim
from ragas.integrations.ag_ui import convert_messages_snapshot, extract_tool_calls
from ag_ui.core import MessagesSnapshotEvent, AssistantMessage, ToolCall as AGToolCall, FunctionCall

snap = MessagesSnapshotEvent(messages=[
    AssistantMessage(id="a1", content="Checking", tool_calls=[
        AGToolCall(id="tc1", function=FunctionCall(name="get_weather", arguments='{"city":"SF"}'))]),
])
msgs = convert_messages_snapshot(snap)
print(msgs)
print("extract:", extract_tool_calls(msgs))

The same handler runs for any MESSAGES_SNAPSHOT event routed through convert_to_ragas_messages(), which run_ag_ui_row() uses (verified with a RunStartedEvent + MessagesSnapshotEvent stream).

Error trace

Actual output of the script above:

[AIMessage(content='Checking', metadata=None, type='ai', tool_calls=[ToolCall(name='unknown_tool', args={})])]
extract: [ToolCall(name='unknown_tool', args={})]

A second, independent run against a two-message snapshot (assistant message + tool result) reproduced the same loss:

AIMessage 'Let me check the weather.' [ToolCall(name='unknown_tool', args={})]
ToolMessage 'Sunny, 72F' None
extracted: [ToolCall(name='unknown_tool', args={})]
A) convert_to_ragas_messages -> [('AIMessage', [ToolCall(name='unknown_tool', args={})])]
B) writer produced FunctionCall -> name='get_weather' arguments='{"city": "SF"}'
B) round-trip tool_calls -> [[ToolCall(name='unknown_tool', args={})]]

Expected behavior

convert_messages_snapshot() should return ToolCall(name='get_weather', args={'city': 'SF'}) for the snapshot above, and the round-trip through convert_messages_to_ag_ui() should preserve the tool name and arguments.

Basis:

  • ag-ui-protocol (checked in 0.1.9 — the minimum allowed by this repo — plus 0.1.21 and 0.1.22) declares class ToolCall(ConfiguredBaseModel): id: str; type: Literal["function"] = "function"; function: FunctionCall and class FunctionCall(ConfiguredBaseModel): name: str; arguments: str. Neither class has a top-level name or args; constructing a flat ToolCall(id=..., name=..., args=...) raises ValidationError: function / Field required.
  • The surrounding code already agrees on the nested layout: convert_messages_to_ag_ui() writes FunctionCall(name=tc.name, arguments=json.dumps(tc.args)) in the same file, and _handle_tool_call_end() parses function.arguments JSON for the streaming path.
  • Running the same fallback logic against a real object of the pinned minimum version (ag-ui-protocol==0.1.9):
ToolCall fields: ['id', 'type', 'function']
FunctionCall fields: ['name', 'arguments']
nested: get_weather '{"city": "SF"}'
ragas-style getattr -> unknown_tool {}

Additional context

  • Related issues/PRs: none found. Searched this repo's issues and PRs (open + closed) for ag_ui, unknown_tool and messages_snapshot — no matches, and no open PR currently touches src/ragas/integrations/ag_ui.py.
  • I'm happy to be assigned, and can open a PR that reads the nested function shape (name plus json.loads of the arguments string, with a fallback when it is not valid JSON).