[Bug]: enforce_model_rate_limits — deployment TPM is never enforced for streaming requests (hidden_params.litellm_model_name is None)

Check for existing issues

• I have searched the existing issues and checked that my issue is not a duplicate.

What happened?

What happened?

With optional_pre_call_checks: ["enforce_model_rate_limits"], the per-deployment TPM limit (litellm_params.tpm) is enforced correctly for non-streaming requests but is never enforced for streaming requests. RPM is unaffected.

Root cause: in ModelRateLimitingCheck.async_log_success_event (litellm/router_utils/pre_call_checks/model_rate_limit_check.py), the TPM counter is only incremented when standard_logging_object["hidden_params"]["litellm_model_name"] is truthy:

model = standard_logging_object.get("hidden_params", {}).get("litellm_model_name")
total_tokens = standard_logging_object.get("total_tokens", 0)
if not model or not total_tokens:
    return
tpm_key = f"{model_id}:{model}:tpm:{current_minute}"

For streaming responses that field comes back None, so the method returns early, the TPM key is never created, the counter stays at 0 and the limit can never be reached.

Note the asymmetry: the check itself (async_pre_call_check) does run for streaming — a streaming request is correctly rejected if the counter was already primed by a non-streaming request. Only the write side is missing. The Router's own counter (RouterCacheEnum.TPM → global_router:{id}:{model}:tpm:{minute}) does increment on streaming, because it takes the model name from the local deployment_name variable instead of from the logging payload.

Evidence — the logging payload differs between modes

A CustomLogger registered on the same Router, same model, same prompt:

mode	total_tokens	hidden_params.litellm_model_name	model_id
non-streaming	17	'openai/<model>'	'probe'
streaming	8	None	'probe'

total_tokens and model_id are both present for streaming. Only the model name is missing.

Minimal reproduction

Suggested fix

In ModelRateLimitingCheck.async_log_success_event / log_success_event, fall back to the deployment's model name when hidden_params.litellm_model_name is missing, e.g. resolve it from model_id via the router, or reuse the same value the pre-call check used to build the key (deployment["litellm_params"]["model"]). Populating hidden_params.litellm_model_name for streaming responses would fix it at the source and likely help other consumers too.

Related

#17705 reports the same user-visible symptom (deployment tpm not applied while rpm works) and was closed as not planned. This report adds the underlying cause and a minimal reproduction.

Relevant log output

Not an exception — the limit silently never triggers. The logging payload diff above is the signal.

Are you a ML Ops Team?

Yes

What LiteLLM version are you on?

v1.92.0 — the relevant code in main at the time of writing is unchanged (same cache key, same local_only=True read, same if not model or not total_tokens: return guard), so this is expected to reproduce on current versions as well

User Flow

Single process, single Router, tpm: 100, fresh minute, stream_options={"include_usage": True}:

non-streaming (works):

#	result	limiter counter
1–3	200	45 → 90 → 135
4–6	RateLimitError 429	135 (limit 100 reached)

streaming (not enforced):

#	result	cache keys present
1	200	global_router:probe:openai/<model>:tpm:<min> = 43
2	200	global_router:... = 86
3	200	global_router:... = 129
4	200	global_router:... = 172
5	200	global_router:... = 215

215 tokens consumed against a limit of 100, with no rejection. The limiter's own key (probe:openai/<model>:tpm:<min>) is never created in the streaming run, while it exists in the non-streaming run.

This rules out the usual suspects: it is a single process (not a multi-replica counter split), include_usage is set, the tokens are accounted for (the Router counter increments by the exact amount), and the same limiter rejects correctly in non-streaming mode.

Proof the bug occurs

import asyncio
import litellm
from litellm import Router
from litellm.integrations.custom_logger import CustomLogger

API_BASE = "<your OpenAI-compatible base url>"
API_KEY = "<key or 'no-key-needed'>"
MODEL = "openai/<model served by that endpoint>"

payloads = []


class Probe(CustomLogger):
    async def async_log_success_event(self, kwargs, response_obj, start_time, end_time):
        slo = kwargs.get("standard_logging_object") or {}
        payloads.append(
            {
                "total_tokens": slo.get("total_tokens"),
                "litellm_model_name": (slo.get("hidden_params") or {}).get("litellm_model_name"),
                "model_id": slo.get("model_id"),
            }
        )


litellm.callbacks.append(Probe())

router = Router(
    model_list=[
        {
            "model_name": "g",
            "litellm_params": {"model": MODEL, "api_base": API_BASE, "api_key": API_KEY, "tpm": 100},
            "model_info": {"id": "probe"},
        }
    ],
    optional_pre_call_checks=["enforce_model_rate_limits"],
)


def tpm_keys():
    cache = getattr(router.cache.in_memory_cache, "cache_dict", {}) or {}
    return {k: v for k, v in cache.items() if ":tpm:" in k}


async def call(i, stream):
    kwargs = dict(
        model="g",
        messages=[{"role": "user", "content": f"write one short sentence about the number {i}"}],
        max_tokens=25,
    )
    if stream:
        kwargs.update(stream=True, stream_options={"include_usage": True})
    try:
        resp = await router.acompletion(**kwargs)
        if stream:
            async for _ in resp:
                pass
        return "200"
    except Exception as exc:
        return type(exc).__name__


async def main():
    for mode in (False, True):
        print(f"--- {'streaming' if mode else 'non-streaming'} (tpm=100) ---")
        for i in range(1, 7):
            status = await call(i, mode)
            await asyncio.sleep(2)
            print(f"  {i}: {status:18} keys={tpm_keys()}")
        print(f"  last logging payload: {payloads[-1]}")


asyncio.run(main())

Expected: the streaming loop starts rejecting with RateLimitError once ~100 tokens are consumed. Actual: every streaming call succeeds, the limiter key is never created, and the last payload shows litellm_model_name: None.

What part of LiteLLM is this about?

Proxy

What LiteLLM version are you on ?

v1.92.0

Twitter / LinkedIn details

No response