createEmbeddingContext failure doesn't fall back to CPU the way GPU backend init does

createEmbeddingContext failure doesn't fall back to CPU the way GPU backend init does

Summary

When the GPU backend fails to initialize entirely (no Vulkan/CUDA driver at all), ensureLlama() catches it and retries on CPU, and that path works as designed. But when the backend initializes fine (GPU genuinely present and detected) and a context later fails to allocate, in my case because another process was already holding the VRAM, ensureEmbedContexts() doesn't apply the same fallback. It just throws Failed to create any embedding context and qmd embed/vsearch/query fail outright, even though CPU-mode embedding works fine with the identical model/config.

Environment

• qmd 2.5.3 (@tobilu/qmd)
• Linux x86_64, kernel 6.8.0-138-generic, Vulkan 1.3.275
• Two GPUs on one host: AMD Radeon RX 6900 XT (RADV NAVI21, 16GB) and NVIDIA RTX 3060 (12GB)
• Both GPUs are running a separate Ollama process at the same time, serving LLM inference. VRAM was ~1.3GB free of 28.2GB total when the failure happened
• qmd doctor's device probe reports GPU as available/offloading-enabled (it checks backend presence, not live free VRAM), so nothing in qmd doctor's output predicts the failure

Repro

$ qmd doctor
✓ device probe: GPU vulkan; offloading enabled; devices: AMD Radeon RX 6900 XT (RADV NAVI21), NVIDIA GeForce RTX 3060; VRAM 1.3 GB free / 28.2 GB total; 8 CPU math cores
...
⚠ embedding vector sample: Failed to create any embedding context; rebuild with `qmd embed --force`

$ qmd embed --force
Force re-indexing: clearing all vectors...
Model: embeddinggemma-300M-Q8_0.gguf
Failed to create any embedding context

$ qmd embed --force --no-gpu    # identical model/config, CPU-only
Model: embeddinggemma-300M-Q8_0.gguf
QMD Warning: no GPU acceleration, running on CPU (slow). Run 'qmd doctor' for device diagnostics.
██████████████████████████████ 100%
✓ Done! Embedded 5888 chunks from 1442 documents in 30m 6s

The GPU-mode failure is not flaky. It reproduces every time while the other process holds VRAM, and CPU mode succeeds every time.

Root cause (dist/llm.js, 2.5.3)

ensureLlama() already has the right pattern, one layer above where the failure actually happens:

// llm.js:521 ensureLlama()
catch (err) {
    // GPU backend (e.g. Vulkan/CUDA on headless/driverless machines) can throw at init.
    // Fall back to CPU so qmd still works, and cache the failure to avoid repeated
    // expensive native build/probe attempts in this process.
    failedGpuInitModes.add(gpuMode);
    process.stderr.write(`QMD Warning: GPU init failed...falling back to CPU.\n`);
    llama = await loadCpuCompatibleLlama();
}

But ensureEmbedContexts() (llm.js around line 700-723), which runs after ensureLlama() has already succeeded in GPU mode, has no equivalent:

for (let i = 0; i < n; i++) {
    try {
        this.embedContexts.push(await model.createEmbeddingContext({
            contextSize: LlamaCpp.EMBED_CONTEXT_SIZE,
            ...(threads > 0 ? { threads } : {}),
        }));
    }
    catch {
        if (this.embedContexts.length === 0)
            throw new Error("Failed to create any embedding context");
        break;
    }
}

When the first context creation throws (e.g. vkAllocateMemory failing because another process holds the VRAM), this just rethrows fatally. It never retries with gpuLayers: 0, even though modelLoadOptions() (llm.js:613) already knows how to do that for the forced-CPU path.

Suggested fix

In the catch block around llm.js:719-722, when embedContexts.length === 0 and GPU mode was in effect, retry once by reloading the model with gpuLayers: 0 (or creating the context against a CPU-mode model instance) before giving up. That mirrors ensureLlama()'s existing GPU-init fallback, just one layer deeper. Right now the only escape hatch is a static, whole-process QMD_FORCE_CPU=1/--no-gpu decided in advance. There's no way to try GPU and fall back to CPU per invocation, which matters specifically on a host where GPU is real but dynamically contended by another process rather than structurally absent.

Why this matters beyond my setup

qmd doctor's device probe checks whether a GPU backend is present, not whether VRAM is actually free right now. Any host running qmd alongside another GPU-resident service (a local LLM server, in my case, but this applies to anything holding VRAM) will get a passing doctor report and then a hard failure on the first real embed/search call, with no automatic recovery. That's exactly the gap ensureLlama()'s own fallback comment is meant to avoid ("Fall back to CPU so qmd still works"), just not carried through to the context-creation call site.

Possibly related to the resource-contention angle of #942 ("concurrent invocations exhaust resources"), though that one is about multiple qmd processes competing with each other rather than qmd competing with a non-qmd GPU tenant. Flagging in case there's a shared root cause worth looking at together.