#1565·ggml

libggml isn't throwing an error if intel gpu fence timeout happens but DOES if GGML_VK_PERF_LOGGER=1

Author: erikarnCreated Jul 15, 2026Updated Jul 15, 2026

hi!

This is related to https://github.com/ggml-org/ggml/issues/1554 . What's interesting here is that although i see there's a bunch of fence checks in the vulkan backend, somehow they're not being triggered normally.

Unless! I run llama-cpp or a test case with 'env GGML_VK_PERF_LOGGER=1' to enable performance logging.

Then I can get this (with qwen 3.0, f16):

env GGML_VK_PERF_LOGGER=1 ./llama-cli -m /data/1/adrian/ollama/models/blobs/sha256-bc2421370aa09f86eedd8e57e5b5ff3d1200f4638623c749c2ec820c3b5bff97 -ngl 999 --device Vulkan0 -c 32768 -t 6 -tb 6 -n 512 --flash-attn auto -b 2048 -ub 2048

...

> /read /home/adrian/ai/coder.md.tpl
Loaded text from '/home/adrian/ai/coder.md.tpl'

> hi!
----------------
Vulkan Timings:
ADD: 54 x 1200.86 us = 64846.6 us
FLASH_ATTN_EXT dst(128,16,2048,1),  q(128,2048,16,1),  k(128,2048,8,1),  v(128,2048,8,1),  m(2048,2048,1,1): 28 x 270463 us = 7.57297e+06 us (127.04 GFLOPS/s)
GLU: 27 x 4752.51 us = 128318 us
MUL_MAT f16 m=1024 n=2048 k=1024: 56 x 36387.1 us = 2.03768e+06 us (117.978 GFLOPS/s)
MUL_MAT f16 m=1024 n=2048 k=2048: 28 x 72329.2 us = 2.02522e+06 us (118.733 GFLOPS/s)
MUL_MAT f16 m=1024 n=2048 k=3072: 27 x 107983 us = 2.91555e+06 us (119.304 GFLOPS/s)
MUL_MAT f16 m=2048 n=2048 k=1024: 28 x 72619.8 us = 2.03335e+06 us (118.229 GFLOPS/s)
MUL_MAT f16 m=3072 n=2048 k=1024: 54 x 107833 us = 5.82298e+06 us (119.431 GFLOPS/s)
RMS_NORM_MUL RMS_NORM(1024,2048,1,1): 55 x 2975.51 us = 163653 us
RMS_NORM_MUL_ROPE RMS_NORM(128,16,2048,1): 28 x 42239.7 us = 1.18271e+06 us
RMS_NORM_MUL_ROPE_VIEW_SET_ROWS RMS_NORM(128,8,2048,1): 28 x 21153.1 us = 592287 us
SET_ROWS: 28 x 1427.89 us = 39981.1 us
Total time: 2.45795e+07 us.
ggml_vulkan: ctx->device->device.getQueryPoolResults(ctx->query_pool, 0, ctx->query_idx, (cgraph->n_nodes + 1)*sizeof(uint64_t), timestamps.data(), sizeof(uint64_t), vk::QueryResultFlagBits::e64 | vk::QueryResultFlagBits::eWait) error ErrorDeviceLost at /home/adrian/ai/llama.cpp/ggml/src/ggml-vulkan/ggml-vulkan.cpp:16613

It's tripping up here in ggml-vulkan.cpp :

        // Get the results and pass them to the logger
        std::vector<uint64_t> timestamps(cgraph->n_nodes + 1);
        VK_CHECK(ctx->device->device.getQueryPoolResults(ctx->query_pool, 0, ctx->query_idx, (cgraph->n_nodes + 1)*sizeof(uint64_t), timestamps.data(), sizeof(uint64_t), vk::QueryResultFlagBits::e64 | vk::QueryResultFlagBits::eWait), "get timestamp results");

Is there a way to actually get that fence timeout to trigger /a/ fence that can hit "ErrorDeviceLost" ? Or are things being submitted in a way that prevents you from being able to catch that?

Source: ggml-org/ggml

View original on GitHub View discussion on GitHub