_build_strands_history replays a tool result's media content as raw text, undoing the image/document wrapper the same file builds when first emitting it
Summary
When a Strands agent's tool result contains an image (or document/video), agent.py's outbound path correctly preserves it as a structured {"image": {"format": ..., "source": {"bytes": ...}}}-shaped block wrapped in the ToolMessage.content string (_extract_tool_result_data / _serialize_tool_result_data, lines 1835-1926) - this is explicitly documented as intentional: "Media blocks retain their wrapper ... so the payload type survives conversion to AG-UI's string-only result field."
But _build_strands_history - the function that converts a replayed RunAgentInput.messages history back into Strands-native messages for the next turn - never reverses that. Every historical ToolMessage is unconditionally flattened to plain text:
# https://github.com/ag-ui-protocol/ag-ui/blob/main/integrations/aws-strands/python/src/ag_ui_strands/agent.py#L2239-L2250
pending_tool_results.append(
{
"toolResult": {
"toolUseId": tool_call_id,
"content": [{"text": _coerce_text(msg.content)}],
...
}
}
)_coerce_text (L1718) just does str(content) - it has no awareness that msg.content may itself be the JSON string this same module produced for an image/document block. There's no code path anywhere in _build_strands_history that re-parses that JSON and reconstructs a native {"image": {...}} / {"document": {...}} content block, the way convert_agui_content_to_strands does for user-message media (which is replay-safe, handling image/document/video correctly on every call).
Impact
Any conversation where a tool returns an image is fine on the turn it happens (native image block, cheap tokenization), but on every subsequent turn, the entire base64 payload gets replayed into the model as literal text - a 1.6MB PNG is ~2.2M base64 characters, which tokenizes far worse as text than the same bytes would as a native image block. In our case this alone pushed token usage over a million after a single image and two follow-up turns.
Repro
- Build a Strands agent with an MCP tool that returns
mcp.types.ImageContent. - Call it once through an AG-UI
HttpAgent- the resultingToolMessage.contentcorrectly contains{"image": {"format": "png", "source": {"bytes": "<base64>"}}}. - Send a second message in the same thread, replaying the full message history (the normal AG-UI stateless pattern).
- Inspect what actually reaches the model on turn 2: the tool result arrives as
{"text": "{\"image\": {\"format\": \"png\", \"source\": {\"bytes\": \"<base64>\"}}}"}- the entire base64 string as a literal text block, not a native image.
Suggested fix
In _build_strands_history, when building the toolResult block for a historical ToolMessage, attempt to json.loads(msg.content) and check for the same media-wrapper shapes _extract_tool_result_data produces; if found, emit the corresponding native {"image"/"document"/"video": {...}} content block (base64-decoding source.bytes) instead of {"text": ...}. This would make the library symmetric - what it wraps on the way out, it should be able to unwrap on the way back in - and is the same fix direction convert_agui_content_to_strands already took for user-message media.
Environment
ag-ui-strands0.4.0 (also present onmainas of this report, same line numbers)- Confirmed against a real Bedrock Converse-backed Strands agent
Source: ag-ui-protocol/ag-ui