#20138·lobehub

[Bug] PDF files not rendered to native vision channel when model supports vision

Author: YularzhiCreated Sep 28, 2026Updated Sep 28, 2026

Description

When uploading PDF files (especially scanned PDFs without text layer), the platform does not render pages into images for the native vision channel, even when the chat model has vision capability enabled.

Instead, the platform falls back to the analyzeMedia tool, which requires separate configuration via MULTIMODAL_UNDERSTANDING_PROVIDER and MULTIMODAL_UNDERSTANDING_MODEL environment variables.

This causes extremely slow processing (up to 10 minutes for 2-page PDF) compared to models like Gemini that read PDFs natively in seconds.

Steps to Reproduce

  1. Configure LobeHub with a vision-capable model (e.g., Qwen3.7 Plus with vision: true, GPT-4o, Claude 3.5 Sonnet)
  2. Upload a scanned PDF file (no text layer, e.g., macOS Quartz PDFContext export)
  3. Ask the agent to read/analyze the PDF content
  4. Observe: agent cannot read the PDF, analyzeMedia fails with "Multimodal understanding is not configured"

Expected Behavior

When the chat model has vision: true capability:

  • Platform should render PDF pages into images on the backend
  • Images should be passed directly to the model via native vision channel (image_url in messages)
  • PDF should be read in seconds, similar to how Gemini handles PDFs

Actual Behavior

  • PDF pages are not rendered (empty <page> tags in files_info)
  • Agent falls back to analyzeMedia tool
  • analyzeMedia fails unless MULTIMODAL_UNDERSTANDING_PROVIDER and MULTIMODAL_UNDERSTANDING_MODEL are configured
  • Even with fallback configured, processing is slow (minutes instead of seconds)

Environment

  • LobeHub version: latest (self-hosted)
  • Model: Qwen3.7 Plus (vision: true), GPT-5.6 Luna, etc.
  • Deployment: Docker Compose with PostgreSQL + S3

Technical Details

Current files_info structure for PDF:

xml
<file id="file_xxx" name="scan.pdf" type="application/pdf">
  <page pageNumber="1"></page>
  <page pageNumber="2"></page>
</file>

Pages are empty — not rendered to images.

For regular images (PNG/JPG), native vision works perfectly:

xml
<image ref="msg_xxx.image_1" name="photo.png" url="https://...">

Proposed Solution

  1. Detect vision capability: Check if the active chat model has vision: true
  2. Render PDF pages: If vision is available, render PDF pages to images (e.g., 1400px width, JPEG) on the backend
  3. Pass to native channel: Include rendered pages as image_url in messages, same as regular image uploads
  4. Fallback only when needed: Use analyzeMedia only when model lacks vision capability

Workaround

Currently, users must either:

  • Configure MULTIMODAL_UNDERSTANDING_PROVIDER and MULTIMODAL_UNDERSTANDING_MODEL (duplicates vision capability)
  • Manually convert PDF to images before uploading
  • Use external tools (Gemini, etc.) for PDF reading

Additional Context

This is an architectural issue: the platform treats PDF as a special case requiring separate vision pipeline, instead of leveraging the model's native vision capability. This creates unnecessary complexity and poor UX for PDF processing.