Streaming + tool calling: per-round Usage is lost on 2.0.1 (OpenAI and Anthropic)

Author: almonticCreated Sep 15, 2026Updated Sep 15, 2026

Labelsstatus: waiting-for-triage

Streaming + tool calling: per-round `Usage` is lost on 2.0.1 (OpenAI and Anthropic)

Bug description

On a streaming call that performs a tool call, Spring AI 2.0.1 loses the per-round Usage that 1.1.8 reports. It affects both the OpenAI and the Anthropic modules, with different baselines:

	1.1.8	2.0.1
OpenAI, tool turn	2 rounds, each with full `Usage` incl. `promptTokensDetails.cachedTokens`	1 round, `CompletionUsage` entirely zero, details empty
Anthropic, tool turn	2 rounds, the first with full `Usage`	1 round, `nativeUsage == null`

Without tools, or with tools declared but never called, both versions report correct usage on both providers. The difference appears only when the turn actually calls a tool.

On Anthropic 1.1.8 the second round already reports null, so that part is not a regression; what regresses is the first round, which 1.1.8 reports and 2.0.1 does not.

The impact is on cost accounting: an application that derives cost from the reported tokens loses the input counts — and with them the prompt-cache reads — on every tool-using turn, so cached input ends up charged internally at full price. On our side a flow went from $0.27 to $2.44 per run with no change in the work performed: same model, same number of tool calls, same total input.

Environment

Spring AI 1.1.8 (works) and 2.0.1 (fails) — spring-ai-openai and spring-ai-anthropic, no starter, no Spring Boot. spring-ai-client-chat pinned to the same version: a mixed classpath fails with NoSuchMethodError: ChatOptions.copy() deep inside DefaultChatClient
Java 25
Azure AI Foundry: an OpenAI-compatible endpoint with a gpt-5 class model, and an Anthropic endpoint with a Claude Sonnet class model
No vector store, no chat memory, no advisors
stream_options.include_usage enabled through OpenAiChatOptions.streamUsage(true) (OpenAI only; the Anthropic module streams usage on its own)
Jackson pinned to jackson-annotations 2.21 / jackson-core and jackson-databind 2.21.4 (without this, 2.0.1 fails at runtime with ClassNotFoundException: com.fasterxml.jackson.annotation.JsonSerializeAs)

Steps to reproduce

Build the two projects below, identical except for the model-construction block that each API requires.
Export OPENAI_BASE_URL, OPENAI_API_KEY, OPENAI_MODEL.
Run both. Each runs three scenarios, three identical calls each:

A no tools declared
B tools declared, never called
C tools declared and called

Compare scenario C.

Expected behavior

On a streaming turn that calls a tool, every model call of the chain reports its own Usage, as 1.1.8 does — two rounds, each with its own promptTokens and promptTokensDetails.

Minimal Complete Reproducible example

Common source, identical in both projects:

java

package repro;

import org.springframework.ai.chat.client.ChatClient;
import org.springframework.ai.chat.model.ChatResponse;
import org.springframework.ai.tool.function.FunctionToolCallback;

import java.util.LinkedHashMap;
import java.util.List;
import java.util.Map;
import java.util.function.Function;

public final class Repro {

    /** ~4500 tokens of deterministic filler: the prefix must exceed the 1024-token minimum. */
    private static final String SYSTEM = system();

    private static final String ASK_PLAIN = "Reply with exactly one word: ready.";
    private static final String ASK_TOOL =
            "Call get_weather for Rome, then reply with exactly one word: done.";

    public static void main(String[] args) {
        String model = env("OPENAI_MODEL");
        run("A  no tools declared          ", model, false, ASK_PLAIN);
        run("B  tools declared, not called ", model, true, ASK_PLAIN);
        run("C  tools declared and called  ", model, true, ASK_TOOL);
    }

    private static void run(String label, String model, boolean withTools, String ask) {
        ChatClient client = ChatClient.create(buildModel(model));
        System.out.println(label);
        for (int i = 1; i <= 3; i++) {
            var spec = client.prompt().system(SYSTEM).user(ask);
            if (withTools) {
                spec = spec.toolCallbacks(FunctionToolCallback
                        .builder("get_weather", (Function<Req, String>) r -> "sunny, 21C")
                        .description("Current weather for a city.")
                        .inputType(Req.class)
                        .build());
            }
            // Streaming: on a tool-calling turn the FINAL response carries a cumulative usage
            // with no provider object on BOTH versions, so the per-round numbers must be
            // collected from the chunks as they arrive.
            List<ChatResponse> chunks = spec.stream().chatResponse().collectList().block();
            System.out.printf("   call %d%n", i);
            perRound(chunks).forEach((id, usage) ->
                    System.out.printf("      round %s  %s%n", id, usage));
        }
        System.out.println();
    }

    public record Req(String city) {}

    /** One entry per model call of the turn, keyed by the provider's response id. */
    private static Map<String, String> perRound(List<ChatResponse> chunks) {
        Map<String, String> rounds = new LinkedHashMap<>();
        for (ChatResponse c : chunks == null ? List.<ChatResponse>of() : chunks) {
            if (c == null || c.getMetadata() == null) continue;
            String id = c.getMetadata().getId();
            if (id == null || id.isBlank()) id = "(no id)";
            var u = c.getMetadata().getUsage();
            if (u == null || u.getNativeUsage() == null) {
                rounds.putIfAbsent(id, "prompt=" + (u == null ? "?" : u.getPromptTokens())
                        + "  native=null");
                continue;
            }
            rounds.put(id, "prompt=" + u.getPromptTokens() + "  native=" + u.getNativeUsage());
        }
        return rounds;
    }

    private static String env(String name) {
        String v = System.getenv(name);
        if (v == null || v.isBlank()) throw new IllegalStateException("Missing env " + name);
        return v;
    }

    private static String system() {
        StringBuilder sb = new StringBuilder("You are a test assistant used to measure usage "
                + "reporting. The following paragraphs are filler so that the prefix exceeds the "
                + "minimum cacheable length. They never change.\n\n");
        for (int i = 0; i < 120; i++) {
            sb.append("Paragraph ").append(i).append(": the quick brown fox jumps over the lazy "
                    + "dog while the engineer measures how many tokens the provider reports.\n");
        }
        return sb.toString();
    }
}

Model construction — the only difference between the two projects:

java

// 1.1.8
private static ChatModel buildModel(String model) {
    var api = OpenAiApi.builder()
            .baseUrl(withoutV1(env("OPENAI_BASE_URL")))   // OpenAiApi appends /v1/chat/completions
            .apiKey(env("OPENAI_API_KEY"))
            .build();
    return OpenAiChatModel.builder()
            .openAiApi(api)
            .defaultOptions(OpenAiChatOptions.builder()
                    .streamUsage(true)
                    .model(model)
                    .build())
            .build();
}

// 2.0.1
private static ChatModel buildModel(String model) {
    return OpenAiChatModel.builder()
            .options(OpenAiChatOptions.builder()
                    .baseUrl(withV1(env("OPENAI_BASE_URL")))   // the SDK expects /v1 in the base URL
                    .apiKey(env("OPENAI_API_KEY"))
                    .streamUsage(true)
                    .model(model)
                    .build())
            .build();
}

Dependencies (same in both, only spring-ai.version changes):

xml

<dependency>
  <groupId>org.springframework.ai</groupId>
  <artifactId>spring-ai-openai</artifactId>
  <version>1.1.8</version>   <!-- or 2.0.1 -->
</dependency>
<!-- pinned so both variants run on the same Jackson; 2.0.1 needs annotations that older
     releases do not carry (ClassNotFoundException: JsonSerializeAs) -->
<dependency>
  <groupId>com.fasterxml.jackson.core</groupId>
  <artifactId>jackson-annotations</artifactId><version>2.21</version>
</dependency>
<dependency>
  <groupId>com.fasterxml.jackson.core</groupId>
  <artifactId>jackson-databind</artifactId><version>2.21.4</version>
</dependency>
<dependency>
  <groupId>com.fasterxml.jackson.core</groupId>
  <artifactId>jackson-core</artifactId><version>2.21.4</version>
</dependency>

Output — 1.1.8

A  no tools declared
   call 1
      round chatcmpl-…  prompt=4489  Usage[promptTokens=4489, totalTokens=4493,
                                      promptTokensDetails=[cachedTokens=4486]]

B  tools declared, not called
   call 1
      round chatcmpl-…  prompt=4606  Usage[promptTokens=4606, totalTokens=4610,
                                      promptTokensDetails=[cachedTokens=4603]]

C  tools declared and called
   call 1
      round chatcmpl-EOLdle…  prompt=4613  Usage[promptTokens=4613, totalTokens=4645,
                                           promptTokensDetails=[cachedTokens=4610]]
      round chatcmpl-EOLdjF…  prompt=4613  Usage[promptTokens=4613, totalTokens=4645,
                                           promptTokensDetails=[cachedTokens=4610]]

Output — 2.0.1

A  no tools declared
   call 1
      round chatcmpl-…  prompt=4489  CompletionUsage{promptTokens=4489, totalTokens=4493,
                                     promptTokensDetails=PromptTokensDetails{cachedTokens=4486}}

B  tools declared, not called
   call 1
      round chatcmpl-…  prompt=4606  CompletionUsage{promptTokens=4606, totalTokens=4610,
                                     promptTokensDetails=PromptTokensDetails{cachedTokens=4603}}

C  tools declared and called
   call 1
      round chatcmpl-EOLe6i…  prompt=0  CompletionUsage{completionTokens=0, promptTokens=0,
                                        totalTokens=0, completionTokensDetails=,
                                        promptTokensDetails=, additionalProperties={}}

For the Anthropic variants the model-construction block is the only change again:

java

// 1.1.8 — base URL carries no version segment, the module appends /v1/messages
var api = AnthropicApi.builder()
        .baseUrl(env("ANTHROPIC_BASE_URL"))
        .apiKey(env("ANTHROPIC_API_KEY"))
        .build();
return AnthropicChatModel.builder()
        .anthropicApi(api)
        .defaultOptions(AnthropicChatOptions.builder().model(model).maxTokens(256).build())
        .build();

// 2.0.1 — connection details moved into the options, official Anthropic SDK over OkHttp
return AnthropicChatModel.builder()
        .options(AnthropicChatOptions.builder()
                .baseUrl(env("ANTHROPIC_BASE_URL"))
                .apiKey(env("ANTHROPIC_API_KEY"))
                .model(model)
                .maxTokens(256)
                .build())
        .build();

streamUsage does not exist on the Anthropic options and is not needed: that module streams usage on its own.

Output — Anthropic 1.1.8

A  no tools declared
   call 1
      round msg_…  prompt=7511  Usage[inputTokens=7511, outputTokens=4,
                                cacheCreationInputTokens=0, cacheReadInputTokens=0]

B  tools declared, not called
   call 1
      round msg_…  prompt=7971  Usage[inputTokens=7971, outputTokens=4, …]

C  tools declared and called
   call 1
      round msg_011Cf5C9zn…  prompt=7981   Usage[inputTokens=7981, outputTokens=50, …]
      round msg_011Cf5CA6u…  prompt=16026  native=null

Output — Anthropic 2.0.1

A  no tools declared
   call 1
      round msg_…  prompt=7511  MessageDeltaUsage{inputTokens=7511, outputTokens=5,
                                cacheCreationInputTokens=0, cacheReadInputTokens=0, …}

B  tools declared, not called
   call 1
      round msg_…  prompt=7935  MessageDeltaUsage{inputTokens=7935, outputTokens=4, …}

C  tools declared and called
   call 1
      round msg_011Cf5CBc8…  prompt=15974  native=null

On Anthropic 1.1.8 the tool turn yields two rounds and the first one carries its Usage; on 2.0.1 a single round surfaces and it carries none. Note also that in 2.x the native object is a MessageDeltaUsage (from the message_delta event) rather than the full Usage of 1.x, which may be where the tool-turn path loses it.

Across all four runs, scenarios A and B are directly comparable between the two versions, which rules out the endpoint, the model, the prompt and the mere presence of tool definitions. C is the only difference, on both providers.

Notes for whoever tries to reproduce

A non-streaming reproducer cannot show this: with .call() the final response of a tool turn carries a cumulative usage with no provider object on both versions, so scenario C prints null either way and measures nothing.
streamUsage(true) is required; without it the provider sends no usage at all and every scenario reads zero on both versions.
The OpenAI base URL must be adapted per version (1.1.8 appends /v1/chat/completions, the 2.x SDK expects /v1 already in the base URL). Unrelated to the bug, but needed to run both against one endpoint. The Anthropic base URL is the same on both.
Keep spring-ai-client-chat at the same version as the provider module. A mixed classpath does not fail the measurement, it silently mixes two versions — here it happened to blow up with NoSuchMethodError: ChatOptions.copy(), which is luckier than it sounds.

Source: spring-projects/spring-ai

View original on GitHub View discussion on GitHub