Baike.dev
Log in
Back to tool/Back to issues
#1857·agent-zero

Vision images re-attach as base64 into every LLM request - image-heavy chats hit provider 413 and die silently

Author: nico7masterCreated Aug 28, 2026Updated Sep 13, 2026

Vision images re-attach as base64 into every LLM request — image-heavy chats hit provider 413 and die silently

Summary

Images loaded via vision_load remain in chat history as image_url parts forever. Every subsequent LLM call re-attaches all of them as base64 into the request body. On image-heavy long-running chats the body reaches tens of MB while the context indicator shows only 12–24% token usage. Providers behind Cloudflare (e.g. z.ai) reject the body with 413 Request Entity Too Large; LiteLLM retries 3× with the identical oversized body, all fail, and the turn dies with no visible error — the UI hangs at "Reasoning.../waiting for input" until the user restarts. Affects all presets/endpoints; text compaction does not fix it (the text shrinks, the images stay).

Reproduction

  1. In a long-running chat, vision_load several multi-MB images (e.g. 6 PNGs of ~3 MB each) across several turns
  2. Keep chatting — every turn now sends the accumulated base64 payloads of all historical images in the request body
  3. Once accumulated payload crosses the provider's body limit (observed threshold on a Cloudflare-fronted endpoint at ~25 MB total PNGs → base64-inflated body), every turn fails:
litellm.exceptions.APIError: APIError: OpenAIException - <html>... 413 Request Entity Too Large ...
LiteLLM Retried: 3 times

No error is surfaced in the WebUI; the chat appears silently hung. Real-world correlation from our instance: vision_load at 12:25 → 413 at 12:29; 6 beat frames (~18 MB) loaded 13:58 → turn dead; same chat 413'd again at 14:00 and 15:13 — while token counts showed 12–24%.

Root cause

  1. helpers/history.py:794 treats image_url parts as permanent history items; vision-loaded images (stored as files under chats/<id>/images/vision-load/, referenced by path) are never pruned.
  2. helpers/litellm_transport.py:937-945 (content_part_from_chat) converts every historical image_url part into an input_image part for each request. Local paths are read and base64-encoded by the transport, so N historical images ≈ N × (file size × 1.37) bytes in every single request body.
  3. The context indicator counts tokens, not body bytes — a chat can show 12% context while its request body exceeds the provider's upload limit.
  4. On 413, exception_mapping_utils.py:633 raises a generic APIError; LiteLLM retries 3× with the same body; the failure surfaces only as a generic "Critical error occurred, retrying..." and then a silent dead turn.

Proposed fix

Trim historical image parts from the outgoing request (they serve no purpose after the model has seen them once). We run this as a user-space extension on call_chat_model_turn/start (verified working — a turn against the exact history that 413'd four times completed without error):

  • Keep only the K most recent image parts in the outgoing message list (K=2 works well); replace older ones with a text placeholder "[earlier image removed to keep request size down]"
  • Operate on a copy — never mutate stored history
  • Optionally: convert old images to low-detail thumbnails instead of placeholders, if visual continuity matters

A more conservative alternative: cap the cumulative base64 payload per request (e.g. 10 MB) and drop/replace oldest images until under the cap, logging a warning.

Before/after

Scenario Before After
Chat with many historical vision images Every request re-sends all images; eventual deterministic 413; silent hang Only K recent images sent; body stays bounded
Token indicator Shows 12–24% while body is 25+ MB (misleading) Unchanged (indicator is token-based; a body-size warning would help further)
413 from provider 3 identical retries, silent dead turn Avoided in the first place
Compaction of image-heavy chat Does not help (text shrinks, images stay) Unaffected — images trimmed at request time
Normal image use (recent images) Works Unchanged (kept)

Environment

  • Agent Zero v2.10, Docker (Kali container), Chat Completions mode
  • Provider: z.ai coding endpoint behind Cloudflare (body-size limit observed at roughly 20–30 MB)
  • Reproduced across 4 separate incidents on 2026-08-28 in the same chat

Related: #1855 (Chat Completions mode plain-text logging gap — same "turn dies silently" UX class).

Source: agent0ai/agent-zero

View original on GitHubView discussion on GitHub