Bug: Streaming LLM calls hang indefinitely — no stream_timeout default, retry logic never triggers on stalled streams

Bug: Streaming LLM calls can hang indefinitely — no stream_timeout (or int-typed timeout) by default, retry logic never triggers

Summary

When the LLM provider accepts a streaming request but stalls (never sends the first chunk, or stops sending chunks mid-stream), Agent Zero waits indefinitely — the UI shows the chat as stuck for hours with no error. Two gaps combine:

1. No default timeout / stream_timeout is passed to LiteLLM (DEFAULT_LITELLM_GLOBAL_KWARGS only contains drop_params).
2. The retry loop only retries on exceptions — a hung stream raises none, so it never retries and never aborts.

Observed live against a slow/unstable OpenAI-compatible endpoint: a trivial 5-token request took 107 s; with a large chat context, turns hang for hours with no timeout firing.

Reproduction

1. Point an Agent Zero model preset at an OpenAI-compatible endpoint that stalls (accepts the connection, sends no chunks — e.g. an overloaded proxy).
2. Send a chat message so the agent starts a streaming LLM turn.
3. Observe: models.py enters async for parsed in transport.astream(): and waits forever. No error, no retry, no watchdog cancellation — the chat appears frozen.

Intermittent condition: requires the endpoint to stall after accepting the connection (pure connection-refused fails fast and retries fine). We reproduced this naturally with the llm.agent-zero.ai Venice proxy during load (413 "Request body too large" storms in parallel chats + one measured 107 s response for max_tokens=5).

Root cause

models.py:

DEFAULT_LITELLM_GLOBAL_KWARGS: dict[str, Any] = {
    "drop_params": True,
}

→ No timeout, no stream_timeout defaults. get_litellm_global_kwargs() merges user settings over this, so instances that never configured kwargs get no timeouts at all.

Streaming turn (models.py, stream_call path):

while True:
    got_any_chunk = False
    try:
        if stream:
            async for parsed in transport.astream():   # ← awaits indefinitely on a stalled stream
                ...
    except Exception as e:
        # Retry only if no chunks received and error is transient
        if got_any_chunk or not _is_transient_litellm_error(e) or attempt >= max_retries:
            raise
        attempt += 1
        await asyncio.sleep(retry_delay_s)

→ The retry mechanism is exception-driven. LiteLLM raises timeout errors only if a timeout kwarg is set; mid-stream stalls require stream_timeout. With neither set, a stalled stream is an infinite await, not an exception — so max_retries (default 2) never comes into play.

Additional sharp edge: settings UI/env stores litellm_global_kwargs.timeout as a string (we found "timeout": "120" in usr/settings.json); string values passed through to some providers can behave inconsistently. Normalizing numeric kwargs in _normalize_litellm_kwargs() would harden this.

Note: the 15-second UI watchdog ("Returning control to agent...") frees the agent loop but does not cancel the underlying hung HTTP request.

Proposed fix

Minimal diff — ship safe defaults:

DEFAULT_LITELLM_GLOBAL_KWARGS: dict[str, Any] = {
    "drop_params": True,
    "timeout": 180,        # covers connect + time-to-first-chunk; LiteLLM raises on exceed → existing retry logic applies
    "stream_timeout": 300, # hard cap per stream; aborts mid-stream stalls
}

More conservative alternative: don't change defaults; instead (a) document litellm_global_kwargs timeout keys prominently, and (b) in _normalize_litellm_kwargs(), coerce numeric timeout keys to int/float. Either way, a stalled stream must become a retriable exception rather than an infinite await.

(Verified locally: setting timeout=180, stream_timeout=300 via litellm_global_kwargs flows through _merge_litellm_call_kwargs() into every LiteLLM call — the fix works via config alone, no code change strictly required, but defaults are the actual bug: out-of-the-box instances hang.)

Before / after

Environment

• Agent Zero, Docker container (Debian/Kali base), framework runtime Python 3.12, LiteLLM via framework venv
• Model access via OpenAI-compatible proxy (a0 venice / z-ai glm); behavior is provider-independent — any stalling endpoint triggers it
• First systematically observed 2026-09-06/07; root-caused via live latency measurement (107 s for 5 tokens) and log analysis (exceptions_capture.jsonl: repeated litellm.APIError ... 413 Request body too large in concurrent chats while this chat's stream hung)