Copilot can't see images it didn't receive as attachments (PDF pages, workspace images)
Author: ntindleCreated Sep 23, 2026Updated Oct 1, 2026
Problem
Only user-attached image files reach the model as vision blocks, at the start of a turn (copilot/sdk/service.py:3585, copilot/baseline/service.py:393). Every other image is invisible to the model:
read_workspace_filereturns images as base64 inside JSON text, not as an image content block.- When a tool output is summarized,
copilot/tools/base.py:47replaces anycontent_base64over 1,000 chars with<binary, ~N bytes>, so even the base64 text is gone. - PDF attachments (scanned documents, handwriting) are never shown to the model, and page images the copilot renders in the sandbox can't be viewed either.
- The tool description promises "Small text/image files return inline" (
copilot/tools/workspace_files.py:444), so the model keeps trying.
This is the same on the SDK and baseline paths.
Impact (from copilot trace review)
A production turn asked the copilot to evaluate a scanned exam PDF. It spent ~21 minutes, 46 tool calls and ~$1.50 trying: rendering pages to PNG, the SDK Read tool, read_file (raw bytes), read_workspace_file, shrinking images to ~8 KB (illegible), serving them on a sandbox HTTP server for the browser tool (blocked), and a data: URL (rejected). It ended with "I've hit a hard technical limit". The user was charged for the whole attempt.
Proposal
- Return images from tools as real image content blocks (vision) when under a size limit:
read_workspace_filefor image MIME types,browser_screenshot, and a way to view rendered PDF pages. - Let the copilot request PDF pages as images (e.g. a
pagesargument), or auto-embed small PDF attachments page by page. - Make the
read_workspace_filedescription state exactly what the model receives, and add one line to the system prompt on how to view an image.
Acceptance criteria
- The copilot can view a PNG/JPEG from the workspace (≤ a few MB) and describe it.
- The copilot can read the pages of an attached scanned PDF.
- A regression test covers a tool result carrying an image block through to the model request.
Source: Significant-Gravitas/AutoGPT