[BUG: Breaking] Parallel inference not working on llama.cpp backend.
Author: SarcasticBaka29Created Jul 2, 2026Updated Jul 21, 2026
Labelsbug: breaking
Hey there, I'm trying this project and I can't seem to get parallel processing to work using llama.cpp backend, despite setting the "SURYA_INFERENCE_PARALLEL" environment variable on the client side to values such as 4 or 8 which correspond to "-np" value used when starting llama-server, the actual inference always seems to use a single slot no matter what.
Here's the console log:
0.01.067.790 I srv init: init: chat template, thinking = 0
0.01.067.818 I srv llama_server: model loaded
0.01.067.821 I srv llama_server: listening on http://0.0.0.0:8081
0.01.067.836 I srv update_slots: all slots are idle
0.13.230.185 I srv operator (): chat format: peg-native
0.13.230.311 I slot get_availabl: id 3 | task -1 | selected slot by LRU, t_last = -1
0.13.230.312 I srv get_availabl: updating prompt cache
0.13.230.323 I srv load: - looking for better prompt, base f_keep = -1.000, sim = 0.000
0.13.230.331 I srv update: - cache state: 0 prompts, 0.000 MiB (limits: 8192.000 MiB, 50176 tokens, 8589934592 est)
0.13.230.333 I srv get_availabl: prompt cache update took 0.02 ms
0.13.239.230 I slot launch_slot_: id 3 | task -1 | sampler chain: logits -> ?penalties -> ?dry -> ?top-n-sigma -> top-k -> ?typical -> top-p -> min-p -> ?xtc -> temp-ext -> dist
0.13.239.259 I slot launch_slot_: id 3 | task -1 | sampler params:
repeat_last_n = 64, repeat_penalty = 1.000, frequency_penalty = 0.000, presence_penalty = 0.000
dry_multiplier = 0.000, dry_base = 1.750, dry_allowed_length = 2, dry_penalty_last_n = 12544
top_k = 40, top_p = 0.100, min_p = 0.050, xtc_probability = 0.000, xtc_threshold = 0.100, typical_p = 1.000, top_n_sigma = -1.000, temp = 0.000
mirostat = 0, mirostat_lr = 0.100, mirostat_ent = 5.000, adaptive_target = -1.000, adaptive_decay = 0.900
0.13.239.264 I slot launch_slot_: id 3 | task 0 | processing task, is_child = 0
0.13.239.265 I slot process_sing: id 0 | task -1 | saving idle slot to prompt cache
0.13.239.266 I slot process_sing: id 1 | task -1 | saving idle slot to prompt cache
0.13.239.267 I slot process_sing: id 2 | task -1 | saving idle slot to prompt cache
0.13.239.284 I slot operator (): id 3 | task 0 | new prompt, n_ctx_slot = 12544, n_keep = 0, task.n_tokens = 3364
0.13.239.290 I slot operator (): id 3 | task 0 | cached n_tokens = 0, memory_seq_rm [0, end)
0.13.246.691 I slot operator (): id 3 | task 0 | cached n_tokens = 7, memory_seq_rm [7, end)
0.13.246.768 I slot process_mtmd: id 3 | task 0 | encoding mtmd batch from idx = 7, n_chunks = 1
0.19.628.115 I decoding image batch 1/2, n_tokens_batch = 2048
0.19.630.655 W find_slot: non-consecutive token position 7 after 6 for sequence 3 with 512 new tokens
0.19.630.662 W find_slot: non-consecutive token position 7 after 7 for sequence 3 with 512 new tokens
0.19.630.663 W find_slot: non-consecutive token position 7 after 7 for sequence 3 with 512 new tokens
0.19.630.664 W find_slot: non-consecutive token position 7 after 7 for sequence 3 with 512 new tokens
0.19.631.427 W find_slot: non-consecutive token position 7 after 6 for sequence 3 with 512 new tokens
0.19.644.963 W find_slot: non-consecutive token position 7 after 7 for sequence 3 with 512 new tokens
0.19.979.146 W find_slot: non-consecutive token position 7 after 7 for sequence 3 with 512 new tokens
0.20.208.776 W find_slot: non-consecutive token position 7 after 7 for sequence 3 with 512 new tokens
0.20.519.215 I image decoded (batch 1/2) in 891 ms
0.20.519.236 I decoding image batch 2/2, n_tokens_batch = 1195
0.20.520.864 W find_slot: non-consecutive token position 7 after 7 for sequence 3 with 512 new tokens
0.20.520.871 W find_slot: non-consecutive token position 7 after 7 for sequence 3 with 512 new tokens
0.20.520.872 W find_slot: non-consecutive token position 7 after 7 for sequence 3 with 171 new tokens
0.20.521.360 W find_slot: non-consecutive token position 7 after 7 for sequence 3 with 512 new tokens
0.20.763.394 W find_slot: non-consecutive token position 7 after 7 for sequence 3 with 512 new tokens
0.21.087.385 W find_slot: non-consecutive token position 7 after 7 for sequence 3 with 171 new tokens
0.21.342.369 I image decoded (batch 2/2) in 823 ms
0.21.349.680 W find_slot: non-consecutive token position 185 after 7 for sequence 3 with 110 new tokens
0.21.349.831 W find_slot: non-consecutive token position 185 after 7 for sequence 3 with 110 new tokens
0.21.449.347 I slot print_timing: id 3 | task 0 | prompt processing, n_tokens = 3360, progress = 1.00, t = 8.21 s / 409.25 tokens per second
0.21.449.358 I slot operator (): id 3 | task 0 | cached n_tokens = 3360, memory_seq_rm [186, end)
0.21.449.502 I slot init_sampler: id 3 | task 0 | init sampler, took 0.04 ms, tokens: text = 121, total = 3364
0.22.364.127 I slot create_check: id 3 | task 0 | created context checkpoint 1 of 32 (pos_min = 185, pos_max = 185, n_tokens = 3360, size = 19.266 MiB)
0.25.398.577 I slot print_timing: id 3 | task 0 | n_decoded = 166, tg = 55.28 t/s, tg_3s = 55.27 t/s
0.28.406.494 I slot print_timing: id 3 | task 0 | n_decoded = 332, tg = 55.23 t/s, tg_3s = 55.19 t/s
0.31.423.007 I slot print_timing: id 3 | task 0 | n_decoded = 498, tg = 55.16 t/s, tg_3s = 55.03 t/s
0.34.432.459 I slot print_timing: id 3 | task 0 | n_decoded = 661, tg = 54.91 t/s, tg_3s = 54.16 t/s
0.37.445.120 I slot print_timing: id 3 | task 0 | n_decoded = 826, tg = 54.89 t/s, tg_3s = 54.77 t/s
0.40.449.338 I slot print_timing: id 3 | task 0 | n_decoded = 990, tg = 54.84 t/s, tg_3s = 54.59 t/s
0.43.450.759 I slot print_timing: id 3 | task 0 | n_decoded = 1154, tg = 54.81 t/s, tg_3s = 54.64 t/s
0.46.457.425 I slot print_timing: id 3 | task 0 | n_decoded = 1318, tg = 54.78 t/s, tg_3s = 54.55 t/s
0.49.461.963 I slot print_timing: id 3 | task 0 | n_decoded = 1482, tg = 54.75 t/s, tg_3s = 54.58 t/s
0.52.474.282 I slot print_timing: id 3 | task 0 | n_decoded = 1646, tg = 54.72 t/s, tg_3s = 54.44 t/s
0.55.480.499 I slot print_timing: id 3 | task 0 | n_decoded = 1809, tg = 54.68 t/s, tg_3s = 54.22 t/s
0.58.496.482 I slot print_timing: id 3 | task 0 | n_decoded = 1973, tg = 54.65 t/s, tg_3s = 54.38 t/s
1.01.510.962 I slot print_timing: id 3 | task 0 | n_decoded = 2136, tg = 54.61 t/s, tg_3s = 54.07 t/s
1.04.514.337 I slot print_timing: id 3 | task 0 | n_decoded = 2298, tg = 54.56 t/s, tg_3s = 53.94 t/s
1.07.516.604 I slot print_timing: id 3 | task 0 | n_decoded = 2460, tg = 54.52 t/s, tg_3s = 53.96 t/s
1.10.519.104 I slot print_timing: id 3 | task 0 | n_decoded = 2621, tg = 54.46 t/s, tg_3s = 53.62 t/s
1.13.523.252 I slot print_timing: id 3 | task 0 | n_decoded = 2783, tg = 54.43 t/s, tg_3s = 53.93 t/s
1.16.536.062 I slot print_timing: id 3 | task 0 | n_decoded = 2944, tg = 54.38 t/s, tg_3s = 53.44 t/s
1.19.541.100 I slot print_timing: id 3 | task 0 | n_decoded = 3105, tg = 54.33 t/s, tg_3s = 53.58 t/s
1.22.548.659 I slot print_timing: id 3 | task 0 | n_decoded = 3266, tg = 54.29 t/s, tg_3s = 53.53 t/s
1.25.203.290 I slot print_timing: id 3 | task 0 | prompt eval time = 9155.92 ms / 3364 tokens ( 2.72 ms per token, 367.41 tokens per second)
1.25.203.300 I slot print_timing: id 3 | task 0 | eval time = 62807.72 ms / 3408 tokens ( 18.43 ms per token, 54.26 tokens per second)
1.25.203.301 I slot print_timing: id 3 | task 0 | total time = 71963.64 ms / 6772 tokens
1.25.203.302 I slot print_timing: id 3 | task 0 | graphs reused = 3394
1.25.205.211 I slot release: id 3 | task 0 | stop processing: n_tokens = 6771, truncated = 0
1.25.206.999 I srv update_slots: all slots are idle
1.25.759.758 I srv stream_sessi: conv_id= (empty=1) Source: datalab-to/surya