[Bug] PDF files not rendered to native vision channel when model supports vision
Description
When uploading PDF files (especially scanned PDFs without text layer), the platform does not render pages into images for the native vision channel, even when the chat model has vision capability enabled.
Instead, the platform falls back to the analyzeMedia tool, which requires separate configuration via MULTIMODAL_UNDERSTANDING_PROVIDER and MULTIMODAL_UNDERSTANDING_MODEL environment variables.
This causes extremely slow processing (up to 10 minutes for 2-page PDF) compared to models like Gemini that read PDFs natively in seconds.
Steps to Reproduce
- Configure LobeHub with a vision-capable model (e.g., Qwen3.7 Plus with
vision: true, GPT-4o, Claude 3.5 Sonnet) - Upload a scanned PDF file (no text layer, e.g., macOS Quartz PDFContext export)
- Ask the agent to read/analyze the PDF content
- Observe: agent cannot read the PDF,
analyzeMediafails with "Multimodal understanding is not configured"
Expected Behavior
When the chat model has vision: true capability:
- Platform should render PDF pages into images on the backend
- Images should be passed directly to the model via native vision channel (image_url in messages)
- PDF should be read in seconds, similar to how Gemini handles PDFs
Actual Behavior
- PDF pages are not rendered (empty
<page>tags in files_info) - Agent falls back to
analyzeMediatool analyzeMediafails unlessMULTIMODAL_UNDERSTANDING_PROVIDERandMULTIMODAL_UNDERSTANDING_MODELare configured- Even with fallback configured, processing is slow (minutes instead of seconds)
Environment
- LobeHub version: latest (self-hosted)
- Model: Qwen3.7 Plus (vision: true), GPT-5.6 Luna, etc.
- Deployment: Docker Compose with PostgreSQL + S3
Technical Details
Current files_info structure for PDF:
<file id="file_xxx" name="scan.pdf" type="application/pdf">
<page pageNumber="1"></page>
<page pageNumber="2"></page>
</file>Pages are empty — not rendered to images.
For regular images (PNG/JPG), native vision works perfectly:
<image ref="msg_xxx.image_1" name="photo.png" url="https://...">Proposed Solution
- Detect vision capability: Check if the active chat model has
vision: true - Render PDF pages: If vision is available, render PDF pages to images (e.g., 1400px width, JPEG) on the backend
- Pass to native channel: Include rendered pages as image_url in messages, same as regular image uploads
- Fallback only when needed: Use
analyzeMediaonly when model lacks vision capability
Workaround
Currently, users must either:
- Configure
MULTIMODAL_UNDERSTANDING_PROVIDERandMULTIMODAL_UNDERSTANDING_MODEL(duplicates vision capability) - Manually convert PDF to images before uploading
- Use external tools (Gemini, etc.) for PDF reading
Additional Context
This is an architectural issue: the platform treats PDF as a special case requiring separate vision pipeline, instead of leveraging the model's native vision capability. This creates unnecessary complexity and poor UX for PDF processing.
Source: lobehub/lobehub