[BUG: Breaking] Inference error with llama.cpp backend: failed to parse grammar due to \d escape in bbox schema
Describe the Bug
When running Surya with the llamacpp inference backend and the GGUF model datalab-to/surya-ocr-2-gguf, inference fails immediately with an OpenAI-compatible API error:
Error code: 400 - {'error': {'code': 400, 'message': 'Failed to initialize samplers: failed to parse grammar', 'type': 'invalid_request_error'}}
The llama-server log shows that grammar parsing fails on the generated response format grammar for bbox:
response-format-schema-item-bbox ::= "\"" ("\d"{1,4} " \d"{1,4} " \d"{1,4} " \d"{1,4}) "\""
It looks like llama.cpp grammar parsing rejects \d as an unknown escape. https://github.com/ggml-org/llama.cpp/issues/16714
Input Document
The issue happens during inference on a PDF/input document.
Output Trace / Stack Trace
llama-server log:
Click to expand0.00.024.897 I cmn common_param: common_params_print_info: verbosity = 3 (adjust with the `-lv N` CLI arg)
0.00.026.636 I srv load_model: loading model '/root/.cache/huggingface/hub/models--datalab-to--surya-ocr-2-gguf/snapshots/6a3a4c30e5e74446d4f8b6afd05b2f2da970f470/surya-2.gguf'
0.00.382.722 W load: control-looking token: 1132 '</s>' was not control-type; this is probably a bug in the model. its type will be overridden
0.07.570.133 W load_hparams: Qwen-VL models require at minimum 1024 image tokens to function correctly on grounding tasks
0.07.570.155 W load_hparams: if you encounter problems with accuracy, try adding --image-min-tokens 1024
0.08.603.097 I srv load_model: loaded multimodal model, '/root/.cache/huggingface/hub/models--datalab-to--surya-ocr-2-gguf/snapshots/6a3a4c30e5e74446d4f8b6afd05b2f2da970f470/surya-2-mmproj.gguf'
0.08.789.207 I srv load_model: initializing, n_slots = 8, n_ctx_slot = 12288, kv_unified = 'false'
0.08.793.063 I srv llama_server: model loaded
0.08.793.090 I srv llama_server: listening on http://127.0.0.1:48631
parse: error parsing grammar: unknown escape at \d"{1,4} " \d"{1,4} " \d"{1,4} " \d"{1,4}) "\""
response-format-schema-item-bbox ::= "\"" ("\d"{1,4} " \d"{1,4} " \d"{1,4} " \d"{1,4}) "\""
response-format-schema-item-bbox-kv ::= "\"bbox\"" space ":" space response-format-schema-item-bbox
0.10.296.155 E failed to parse grammar
0.10.296.213 E srv send_error: task id = 0, error: Failed to initialize samplers: failed to parse grammar
0.10.296.228 E srv process_sing: failed to launch slot with task, id_task = 0
0.10.296.246 W srv stop: cancel task, id_task = 0⚙️ Environment
Please fill in all relevant details:
- Marker version: 2.0.0
- Surya version: 0.20.0+
- Python version: 3.11
- PyTorch version: 2.12.0
- Transformers version: 5.14.1
- Operating System (incl. container info if relevant): wsl2
✅ Expected Behavior
The failure appears to be caused by the generated grammar using \d in:
Possibly the grammar should use [0-9] instead of \d for llama.cpp compatibility.
Additional Context
I have also built llama.cpp with llguidance enabled, but the same error still occurs.
Source: datalab-to/surya