#14827·AutoGPT

Copilot can't see images it didn't receive as attachments (PDF pages, workspace images)

Author: ntindleCreated Sep 23, 2026Updated Oct 1, 2026

Problem

Only user-attached image files reach the model as vision blocks, at the start of a turn (copilot/sdk/service.py:3585, copilot/baseline/service.py:393). Every other image is invisible to the model:

  • read_workspace_file returns images as base64 inside JSON text, not as an image content block.
  • When a tool output is summarized, copilot/tools/base.py:47 replaces any content_base64 over 1,000 chars with <binary, ~N bytes>, so even the base64 text is gone.
  • PDF attachments (scanned documents, handwriting) are never shown to the model, and page images the copilot renders in the sandbox can't be viewed either.
  • The tool description promises "Small text/image files return inline" (copilot/tools/workspace_files.py:444), so the model keeps trying.

This is the same on the SDK and baseline paths.

Impact (from copilot trace review)

A production turn asked the copilot to evaluate a scanned exam PDF. It spent ~21 minutes, 46 tool calls and ~$1.50 trying: rendering pages to PNG, the SDK Read tool, read_file (raw bytes), read_workspace_file, shrinking images to ~8 KB (illegible), serving them on a sandbox HTTP server for the browser tool (blocked), and a data: URL (rejected). It ended with "I've hit a hard technical limit". The user was charged for the whole attempt.

Proposal

  1. Return images from tools as real image content blocks (vision) when under a size limit: read_workspace_file for image MIME types, browser_screenshot, and a way to view rendered PDF pages.
  2. Let the copilot request PDF pages as images (e.g. a pages argument), or auto-embed small PDF attachments page by page.
  3. Make the read_workspace_file description state exactly what the model receives, and add one line to the system prompt on how to view an image.

Acceptance criteria

  • The copilot can view a PNG/JPEG from the workspace (≤ a few MB) and describe it.
  • The copilot can read the pages of an attached scanned PDF.
  • A regression test covers a tool result carrying an image block through to the model request.

Source: Significant-Gravitas/AutoGPT