[Bug]: OpenAIChatFormatter: multimodal tool results promoted mid-batch break parallel tool_calls (400 on DeepSeek & other strict OpenAI-compatible providers)

Prerequisites

• I have searched the existing issues and discussions, and this is not a duplicate.
• This is a bug, not a usage question. (For questions, please use Discussions instead.)

Background / Description

Summary

When an agent issues parallel tool calls (multiple ToolCallBlocks in one Msg) and the tool results contain multimodal data (DataBlock, e.g. images read via a read_file tool), OpenAIChatFormatter inserts the promoted multimodal message (role=user, name=system-reminder) between the sibling tool messages of the same tool_calls batch. Providers that strictly validate the protocol reject the whole request with HTTP 400:

An assistant message with 'tool_calls' must be followed by tool messages
responding to each 'tool_call_id'. (insufficient tool messages following
tool_calls message)

Single tool calls are unaffected (the promo naturally lands after the only tool message), so the symptom appears as intermittent 400s on an otherwise healthy session — every "parallel tool call + non-final tool returns an image" turn fails.

Actual output — note the user promo between tool call_A and tool call_B: user - assistant tool_calls: ['call_A', 'call_B'] tool call_A user system-reminder [image promo] tool call_B user system-reminder [image promo]

Sending this history to api.deepseek.com returns: HTTP 400 {"error":{"message":"An assistant message with 'tool_calls' must be followed by tool messages responding to each 'tool_call_id'. (insufficient tool messages following tool_calls message)", "type":"invalid_request_error","code":"invalid_request_error"}}

HTTP 400 {"error":{"message":"An assistant message with 'tool_calls' must be followed by tool messages responding to each 'tool_call_id'. (insufficient tool messages following tool_calls message)", "type":"invalid_request_error","code":"invalid_request_error"}}

Minimal raw-HTTP proof that ordering is the only factor (same payload, same images, same model):

assistant(2 tool_calls) → tool(A) → tool(B) → user(image) → user(image) → 200 OK assistant(2 tool_calls) → tool(A) → user(image) → tool(B) → user(image) → 400 with the error above

Root cause agentscope/formatter/_openai_formatter.py, the ToolResultBlock branch of OpenAIChatFormatter.format (~lines 335–382): each tool result is emitted as its role=tool message, and its multimodal payload is immediately appended as a separate role=user system-reminder message — with no awareness of whether the surrounding tool batch (the preceding assistant tool_calls) still has outstanding responses.

Suggested fix Buffer the promoted multimodal messages while the tool batch is open, and flush them right after the last tool message of that batch, e.g.:

when appending a role=tool message, hold its promo in a per-batch buffer; after appending a tool message, if all tool_call_ids of the preceding assistant message are now answered, flush the buffered promos. Since a well-formed stream can never contain a user message inside an open tool batch, deferring promos to the batch boundary is semantically safe and a no-op for non-multimodal / single-tool flows.

Workaround we currently use Wrap OpenAIChatFormatter.format with a post-pass that defers any user message found while a tool_calls batch still has unanswered ids (pure reordering, content untouched). Happy to contribute the fix upstream if the approach sounds right.

Error Messages

HTTP 400 {"error":{"message":"An assistant message with 'tool_calls' must
be followed by tool messages responding to each 'tool_call_id'.
(insufficient tool messages following tool_calls message)",
"type":"invalid_request_error","code":"invalid_request_error"}}

Steps to Reproduce

Reproduction

import asyncio, base64
from agentscope.formatter import OpenAIChatFormatter
from agentscope.message import (DataBlock, Msg, TextBlock,
                                ToolCallBlock, ToolResultBlock)

IMG = base64.b64encode(b"fake-png").decode()

def image_result(call_id):
    return ToolResultBlock(id=call_id, name="read_file", output=[
        TextBlock(text="[image file]"),
        DataBlock(id=f"data-{call_id}",
                  source={"type": "base64", "data": IMG,
                          "media_type": "image/png"}),
    ])

agent_msg = Msg(name="assistant", role="assistant", content=[
    ToolCallBlock(id="call_A", name="read_file", input='{"path": "a.png"}'),
    ToolCallBlock(id="call_B", name="read_file", input='{"path": "b.png"}'),
    image_result("call_A"),
    image_result("call_B"),
])

async def main():
    messages = await OpenAIChatFormatter().format([
        Msg(name="user", role="user", content=[TextBlock(text="read both")]),
        agent_msg,
    ])
    for m in messages:
        if m["role"] == "user" and m.get("name") == "system-reminder":
            print(m["role"], m["name"], "[image promo]")
        elif m["role"] == "tool":
            print("tool", m["tool_call_id"])
        elif m["role"] == "assistant" and m.get("tool_calls"):
            print("assistant tool_calls:", [tc["id"] for tc in m["tool_calls"]])
        else:
            print(m["role"], "-")

asyncio.run(main())


### Environment

## Environment

- agentscope 2.0.8 (also affects the whole `>=2.0.8,<2.1` range we tested)
- Provider: DeepSeek (`https://api.deepseek.com/chat/completions`) via `OpenAIChatModel` with an OpenAI-compatible credential
- Python 3.13 / 3.14, Windows