Hallucination/generation loop on Blackwell GPUs (sm_120)

Environment

• GPU: NVIDIA RTX PRO 2000 Blackwell (sm_120)
• CUDA: 12.8
• torch: tested with both 2.8.0+cu128 and 2.9.0/2.9.1+cu128
• torchvision: 0.23.0
• transformers: 4.46.3
• tokenizers: 0.20.3
• flash-attn: 2.8.3
• Python: 3.10
• OS: Ubuntu 24.04

Problem

When running DeepSeek-OCR-2 on Blackwell (sm_120) hardware, the model produces severe generation corruption / hallucination loops instead of valid OCR output.

I matched the Hugging Face demo Space environment and inference pipeline as closely as possible, including:

same torch / torchvision / transformers versions same flash-attn wheel same prompt same inference parameters same preprocessing pipeline same model loading code structure

The same image produces correct output on the Hugging Face demo Space

However, locally on Blackwell hardware the model enters unstable repetitive generation loops.

The issue reproduces with both:

torch 2.8.0+cu128 (matching the demo Space dependencies) torch 2.9.0 / 2.9.1+cu128

The documented requirements still reference torch==2.6.0, but this version does not support Blackwell architecture (sm_120). The earliest official prebuilt PyTorch release with Blackwell (sm_120) support is torch 2.7.0+cu128.

Reproduction

Dependencies

torch==2.8.0
torchvision==0.23.0
transformers==4.46.3
tokenizers==0.20.3
accelerate
einops
addict
easydict
flash-attn @ https://github.com/Dao-AILab/flash-attention/releases/download/v2.8.3/flash_attn-2.8.3+cu12torch2.8cxx11abiFALSE-cp310-cp310-linux_x86_64.whl
PyMuPDF
hf_transfer

Using a similar inference as in the demo:

from transformers import AutoModel, AutoTokenizer
from pathlib import Path
from io import StringIO
from PIL import Image, ImageOps
import tempfile
import sys
import torch

IMAGE_PATH = "test.jpg"
MODEL_NAME = "deepseek-ai/DeepSeek-OCR-2"

tokenizer = AutoTokenizer.from_pretrained(
    MODEL_NAME,
    trust_remote_code=True
)

model = AutoModel.from_pretrained(
    MODEL_NAME,
    _attn_implementation="flash_attention_2",
    torch_dtype=torch.bfloat16,
    trust_remote_code=True,
    use_safetensors=True,
)

model = model.eval().cuda()

image = Image.open(IMAGE_PATH)

if image.mode in ("RGBA", "LA", "P"):
    image = image.convert("RGB")

image = ImageOps.exif_transpose(image)

tmp = tempfile.NamedTemporaryFile(delete=False, suffix=".jpg")
image.save(tmp.name, "JPEG", quality=95)
tmp.close()

stdout = sys.stdout
sys.stdout = StringIO()

try:
    res = model.infer(
        tokenizer=tokenizer,
        prompt="<image>\n<|grounding|>Convert the document to markdown.",
        image_file=tmp.name,
        output_path="./output",
        base_size=1024,
        image_size=768,
        crop_mode=True,
        save_results=False,
    )

    captured = sys.stdout.getvalue()
finally:
    sys.stdout = stdout

result = res if res is not None else captured

Path("./output").mkdir(parents=True, exist_ok=True)

with open("./output/result.txt", "w", encoding="utf-8") as f:
    f.write(result)

Expected

Clean markdown OCR output (confirmed working on the HuggingFace demo space with the same image).

Actual

Model enters a hallucination loop, repeating nonsense strings thousands of times and corrupted repetitive token patterns.

Notes

• The model loads correctly and visual tokenization appears normal.

• The issue reproduces even when matching the demo environment very closely.

• The same issue occurs with _attn_implementation="eager", so flash-attn does not appear to be strictly required to trigger it.

• This now appears more like generation instability / decoding corruption rather than ordinary OCR failure.

• At this point the main remaining difference appears to be the runtime/hardware environment, especially Blackwell (sm_120) vs the GPU used by the Hugging Face Space.

• Would appreciate either: (1) confirmation whether DeepSeek-OCR-2 is officially validated on Blackwell/sm_120 GPUs (2) whether bf16 is known to cause instability on Blackwell for this model