Anthropic plugin: hardcoded _NO_PREFILL_PATTERNS misses Claude 4.7/4.8/5, causing HTTP 400 "does not support assistant message prefill"

Version: livekit-agents 1.8.1, livekit-plugins-anthropic 1.8.1 (verified against the released wheels; also present unchanged in 1.5.1, 1.6.8 and 1.7.1)

Related: #6868 (open, covers the model-list half; the tool-pairing caveat below is not addressed there), #6007 (same gap in the AWS plugin), and #5254 / #6213 (both closed, about the content of the injected message rather than which models get one).

Summary

livekit-plugins-anthropic decides whether to append a trailing user message using a hardcoded tuple of two model-name prefixes. Anthropic has since removed assistant-message prefill from a much larger set of models. Any of those newer models therefore takes the "prefill is fine" path, and every request whose context ends on an assistant turn fails with a non-retryable HTTP 400.

Framework behaviour

livekit/plugins/anthropic/llm.py:40-46:

# Claude 4.6+ no longer supports prefilling (trailing assistant messages).
_NO_PREFILL_PATTERNS = ("claude-sonnet-4-6", "claude-opus-4-6")


def _model_disables_prefill(model: str) -> bool:
    """Return True if the model does not support assistant message prefilling."""
    return any(model.startswith(p) for p in _NO_PREFILL_PATTERNS)

livekit/plugins/anthropic/llm.py:220-224:

# Claude 4.6+ does not support prefilling (trailing assistant messages).
inject_trailing = _model_disables_prefill(self._opts.model)
anthropic_ctx, extra_data = chat_ctx.to_provider_format(
    format="anthropic", inject_trailing_user_message=inject_trailing
)

livekit/agents/llm/_provider_format/anthropic.py:105-108:

# Claude 4.6+ does not support prefilling (trailing assistant messages).
# Append a dummy user message so the request ends with a user turn.
if inject_trailing_user_message and messages and messages[-1]["role"] == "assistant":
    messages.append({"role": "user", "content": [{"text": ".", "type": "text"}]})

The comment says "Claude 4.6+", but the tuple enumerates exactly two 4.6 model ids by literal prefix. Anthropic documents prefill as unsupported on Opus 4.6, Sonnet 4.6, Opus 4.7, Opus 4.8, Opus 5, Sonnet 5, and Fable 5 — the tuple covers two of the seven. models.py likewise stops at claude-opus-4-6, so newer ids are only reachable via the str half of model: str | ChatModels, where nothing flags the mismatch.

This tuple is byte-identical in 1.5.1, 1.6.8, 1.7.1 and 1.8.1 — it has not been updated as new models shipped.

Minimal reproduction

No network call needed to see the wrong branch taken:

from livekit.plugins.anthropic.llm import _model_disables_prefill

print(_model_disables_prefill("claude-opus-4-6"))  # True
print(_model_disables_prefill("claude-opus-4-7"))  # False  <-- should be True
print(_model_disables_prefill("claude-opus-4-8"))  # False  <-- should be True
print(_model_disables_prefill("claude-opus-5"))    # False  <-- should be True
print(_model_disables_prefill("claude-sonnet-5"))  # False  <-- should be True

End-to-end, against the API:

from livekit.agents import llm
from livekit.plugins.anthropic import LLM

ctx = llm.ChatContext()
ctx.add_message(role="user", content="hello")
ctx.add_message(role="assistant", content="I was in the middle of")  # trailing assistant

model = LLM(model="claude-opus-4-7")
async for _ in model.chat(chat_ctx=ctx):   # HTTP 400
    pass

Swap the model id to claude-opus-4-6 and the same context succeeds, because the trailing user message is injected.

Expected vs actual

Expected: for any Anthropic model that does not support assistant prefill, to_provider_format is called with inject_trailing_user_message=True, so a context ending on an assistant turn is made legal before dispatch.

Actual: for every model outside the two hardcoded 4.6 prefixes, inject_trailing=False. The array is sent ending on an assistant message and the API returns HTTP 400 (This model does not support assistant message prefill). Because it is a 400, the SDK does not retry — the turn is lost. In a realtime voice agent this surfaces as a dropped response mid-conversation.

This is easy to hit in normal Agent use: generate_reply after a tool call, a greeting emitted after an assistant history item, or any background-triggered reply that copies the context while an assistant turn is trailing.

Suggested fix

Replace the literal-prefix tuple with a pattern that covers the family, so new point releases do not silently regress:

import re

_NO_PREFILL_RE = re.compile(r"^claude-(?:sonnet|opus|haiku|fable)-(?:4-[6-9]|5)")


def _model_disables_prefill(model: str) -> bool:
    return bool(model) and bool(_NO_PREFILL_RE.match(model))

This keeps 4.5 and earlier on the prefill-capable path, where prefill is still valid. Extending ChatModels in models.py to include the current ids would also make the gap visible at type-check time.

One caveat on the fix — trailing tool calls need separate handling

_provider_format/anthropic.py:46-47 maps a function_call item to role="assistant" and emits it at :66-74 as a tool_use block. So a context whose last item is a function_call with no matching function_call_output also serialises to an array ending on an assistant message — and inject_trailing_user_message will happily append a plain-text user message after it.

That produces a different 400. Anthropic's documentation states:

"Tool result blocks must immediately follow their corresponding tool use blocks in the message history. You cannot include any messages between the assistant's tool use message and the user's tool result message."

and names the resulting error:

"tool_use ids were found without tool_result blocks immediately after"

So widening the model list alone converts a prefill 400 into a tool-pairing 400 for contexts with an in-flight tool call. Suggested guard, in to_chat_ctx before the injection at :107:

if inject_trailing_user_message and messages and messages[-1]["role"] == "assistant":
    last_content = messages[-1].get("content") or []
    has_unresolved_tool_use = any(
        isinstance(b, dict) and b.get("type") == "tool_use" for b in last_content
    )
    if not has_unresolved_tool_use:
        messages.append({"role": "user", "content": [{"text": ".", "type": "text"}]})

Callers would still need to drop or resolve the orphan tool_use themselves, but this at least stops the framework from constructing an array that is invalid for a second, less obvious reason. Happy to open a PR for either half.