Nougat returns empty predictions on both full PDF pages and cropped MCQ images (exam-style dataset)

Hello,

I am testing Nougat on a real-world educational dataset consisting of exam-style MCQ PDFs (JEE/CET level), and I encountered a consistent issue where the model produces empty predictions, even though inference runs successfully on GPU.

Dataset Description

The PDF contains:

• Multiple questions per page (Q1–Q50)
• Short question statements (1–3 lines)
• Options in format (a), (b), (c), (d)
• Inline math, vectors (î, ĵ, k̂), and formulas
• Dense, fragmented layout (not paragraph-based)

Environment

• Running on GPU (CUDA enabled)

• Using Nougat inference API

• Input types tested:

  • Full PDF pages → converted to images
  • Cropped individual question images

Observed Behavior

1. Cropped Question Images

For multiple images, model returns:

{'predictions': [''], 'sequences': tensor([[0, 0, 0, ..., 0, 0, 2]], device='cuda:0'), 'repeats': [None], 'repetitions': ['']}

→ Output is completely empty (no extracted text)

2. Full PDF Page Images

Even when processing full pages:

Processing: /temp_images/page_1.png
Processing: /temp_images/page_2.png
...
Done. Total results: 7

Output still:

{'predictions': [''], ...}

→ No meaningful text extracted from entire page

Additional Evidence

• Screenshots show:

  • Proper image loading (Image.open(...).convert("RGB"))
  • Successful inference call (model.inference(image=image))
  • GPU execution (cuda:0)
  • But consistently empty predictions

Key Observations

• Model runs without error (no crash)

• Token sequence is generated but contains mostly zeros → ends early

• Behavior is consistent across:

  • Multiple images
  • Multiple pages
  • Different crops

Hypothesis

Nougat appears to struggle with:

• Fragmented layouts (MCQ format)
• Multiple independent question blocks per page
• Short text segments instead of paragraphs
• Exam-style formatting (Q1, Q2, options)

This is different from:

• Research papers (where Nougat performs well)

Why This Matters

This format is extremely common in:

• Competitive exams (JEE, NEET, CET)
• Question banks
• Educational content digitization
• AI training datasets for education systems

Currently, Nougat is not usable for this type of real-world data.

Suggested Improvements

• Better handling of multi-block page layouts
• Robust extraction from short text segments
• Improved generalization beyond research-style documents
• Avoid empty predictions for valid inputs
• Optional segmentation support for question-based layouts

01. Vectors Hard.pdf

Closing

Nougat is a powerful model for structured scientific documents. Extending support to exam-style MCQ PDFs would unlock significant real-world applications in education.

Happy to provide more examples or datasets if needed.

Best regards, Ajinkya