#58824·vllm

[Bug]: llama3_json streaming drops assistant content that starts with '{' but is not a tool call (non-streaming returns it)

Author: redd34Created Sep 26, 2026Updated Sep 29, 2026

Labelstool-calling

Your current environment

Environment (collect_env.py asserts on macOS, so summarised by hand)

OS: macOS 26.5.1 arm64, Python 3.12.13, CPU only
vLLM: 0.30.0 (pure-Python parser code from the manylinux wheel), and main @ 3b4566c5cf014605de6aeab6eb831b4f20511c17
transformers 5.17.0, tokenizers 0.23.2, torch 2.14.0 (CPU), partial-json-parser 0.2.1.1.post7

Parser-only bug: no GPU, weights or server needed. Tokenizer: unsloth/Llama-3.3-70B-Instruct@99cd0d2c.

Describe the bug

Llama3JsonToolParser.extract_tool_calls_streaming decides between content and tool call from the first character only (current_text.startswith("{") or <|python_tag|>), and never falls back. So streaming and non-streaming disagree on three kinds of plain Llama 3.x output:

A plain answer that starts with { but is not a call (a JSON answer, {}, {"a": 1} is a dict). Non-streaming returns it as content. Streaming returns content=None and no tool calls, so the answer disappears.
A call with empty arguments. Non-streaming gives arguments='{}', streaming gives arguments=''.
Text before the call (Let me check. {"name": ...}; with the default skip_special_tokens=True this is also what a <|python_tag|> call looks like after detokenization). Non-streaming returns the call and drops the text (the docstring says surrounding text is ignored). Streaming returns everything, JSON included, as content, with no tool call.

Case 1 is the most serious one: a client that streams loses the model's whole reply.

python

from vllm.entrypoints.openai.chat_completion.protocol import ChatCompletionRequest
from vllm.tokenizers import get_tokenizer
from vllm.tool_parsers.llama_tool_parser import Llama3JsonToolParser

tok = get_tokenizer("unsloth/Llama-3.3-70B-Instruct",
                    revision="99cd0d2c829e92a67c844f9144c2509632e5c87f")
tools = [{"type": "function", "function": {"name": "get_weather", "parameters": {
    "type": "object", "properties": {"city": {"type": "string"}}}}},
         {"type": "function", "function": {"name": "get_time", "parameters": {
    "type": "object", "properties": {}}}}]
req = ChatCompletionRequest(model="m", messages=[{"role": "user", "content": "hi"}],
                            tools=tools, tool_choice="auto")

def stream(text):
    """One token per delta."""
    ids = tok.encode(text, add_special_tokens=False)
    p = Llama3JsonToolParser(tok, req.tools)
    prev_text, content, calls = "", "", {}
    for i in range(len(ids)):
        cur_text = tok.decode(ids[: i + 1])
        d = p.extract_tool_calls_streaming(prev_text, cur_text, cur_text[len(prev_text):],
                                           ids[:i], ids[: i + 1], ids[i : i + 1], req)
        if d and d.content:
            content += d.content
        for tc in (d.tool_calls if d else []):
            c = calls.setdefault(tc.index, ["", ""])
            c[0] += (tc.function.name or "") if tc.function else ""
            c[1] += (tc.function.arguments or "") if tc.function else ""
        prev_text = cur_text
    return content or None, [tuple(c) for c in calls.values()]

for text in ['{"capital": "Paris", "country": "France"}',
             '{"a": 1} is a dict',
             '{}',
             '{"name": "get_time", "parameters": {}}',
             'Let me check. {"name": "get_weather", "parameters": {"city": "Tokyo"}}']:
    info = Llama3JsonToolParser(tok, req.tools).extract_tool_calls(text, req)
    print(repr(text))
    print("  non-streaming:", info.content, [(t.function.name, t.function.arguments) for t in info.tool_calls])
    print("  streaming    :", *stream(text))

Output on 0.30.0 and on main @ 3b4566c (identical; the KeyError: 'name' log lines are omitted):

'{"capital": "Paris", "country": "France"}'
  non-streaming: {"capital": "Paris", "country": "France"} []
  streaming    : None []
'{"a": 1} is a dict'
  non-streaming: {"a": 1} is a dict []
  streaming    : None []
'{}'
  non-streaming: {} []
  streaming    : None []
'{"name": "get_time", "parameters": {}}'
  non-streaming: None [('get_time', '{}')]
  streaming    : None [('get_time', '')]
'Let me check. {"name": "get_weather", "parameters": {"city": "Tokyo"}}'
  non-streaming: None [('get_weather', '{"city": "Tokyo"}')]
  streaming    : Let me check. {"name": "get_weather", "parameters": {"city": "Tokyo"}} []

Larger deltas (4 tokens, or the whole output in one delta) give the same result for cases 1 and 3.

Expected: streaming returns the same content and tool calls as non-streaming. For case 1, when the buffered {...} turns out not to be a call (no name key, or text after the object), it should be flushed as content. For case 3, the text before the call should be content in both modes, as SGLang's Llama 3.2 detector test expects (test_llama32_detector.py#L84-L88).

Related: I applied the open PR #51577 (llama3_json on the ParserEngine) to main @ 3b4566c, and it fixes all five cases in both modes: content is kept in case 1, '{}' in case 2, and content='Let me check.' plus the call in case 3. So this issue may simply be tracked by that PR. #48294 (whole call in one delta) is a different bug in the same function. sglang#35562 is the same class of bug in SGLang.

Fixtures (template-rendered or taken from the linked tests):

I found this with an automated conformance harness. The report was drafted with AI assistance, and I verified the repro above by running it on both versions.

Found by canitoolcall (https://github.com/redd34/canitoolcall), an offline conformance suite for tool-call parsers.

Before submitting a new issue...

Make sure you already searched for relevant issues, and asked the chatbot living at the bottom right corner of the documentation page, which can answer lots of frequently asked questions.

Source: vllm-project/vllm

View original on GitHub View discussion on GitHub