[BUG]: KVBM disk→device onboarding silently serves the wrong KV when the GDS_MT file registration fails (no nvidia-fs); kvbm 1.4.2 + vLLM 0.27.1 standalone connector

Describe the Bug

With the standalone kvbm==1.4.2 wheel and vLLM 0.27.1 (DynamoConnector, kv_role=kv_both, host + disk tiers), a request whose blocks are onboarded from the disk tier is served with the wrong KV cache — the model answers with the content of the previous (evicting) request — and no error is returned to the client. KVBM's own metrics report a full hit (kvbm_onboard_blocks_d2d == every block of the prompt) while the transfer actually failed:

E gds_mt_backend.cpp:237] GDS_MT: failed to create file handle: GDS_MT: file register error: error=5027, fd=149   (at startup)
E nixl_agent.cpp:921] createXferReq: no specified or potential backend had the required registrations to be able to do the transfer
ERROR dynamo_llm::block_manager::distributed::transfer: Failed to write to blocks: Other(Failed to create XferRequest)
ERROR dynamo_llm::block_manager::distributed::transfer: Batched connector transfer failed: Failed to create XferRequest
ERROR dynamo_runtime::utils::tasks::critical: Critical task 'ZmqActiveMessageWorker: Handler for function: transfer_blocks' failed: Failed to create XferRequest

After that, the engine sits at Running: 0 reqs, Waiting: 1 reqs, Deferred: 1 reqs for minutes (got a finished warning for a request that is onboarding) while /health still returns 200.

The machine has no nvidia-fs kernel module (GB10 / sm_121 aarch64 workstation with unified memory), so cuFile cannot register the disk-cache file. That is an environment limitation; the bug is that (a) build_agent(worker_id, need_disk) in lib/llm/src/block_manager/distributed/worker.rs always creates the GDS_MT backend and the planner uses the direct disk→device path even when the registration failed, without falling back to the POSIX/host two-hop path, and (b) the failed transfer_blocks is not propagated to the connector/vLLM, so the blocks are handed to the scheduler as loaded and the physical GPU blocks still contain the previous request's KV.

The same error string appears in #5012 (closed stale) and in the follow-up reports in #5857.

Steps to Reproduce

Container vllm/vllm-openai:v0.27.1-aarch64-ubuntu2404 (vLLM 0.27.1, torch 2.13 cu130, NIXL 1.3.1), then inside it:

pip install --no-deps kvbm==1.4.2     # plain `pip install kvbm` pulls nixl==1.0.1 and breaks vLLM's nixl_ep import

export DYN_KVBM_CPU_CACHE_GB=8 DYN_KVBM_DISK_CACHE_GB=64 DYN_KVBM_DISK_CACHE_DIR=/kvbm-disk \
       DYN_KVBM_DISABLE_DISK_OFFLOAD_FILTER=true DYN_KVBM_METRICS=true DYN_KVBM_METRICS_PORT=6880

vllm serve Qwen/Qwen3-8B --served-model-name qwen3-8b --port 8000 \
  --max-model-len 40960 --gpu-memory-utilization 0.20 --max-num-seqs 4 --max-num-batched-tokens 8192 \
  --enable-prefix-caching \
  --kv-transfer-config '{"kv_connector":"DynamoConnector","kv_role":"kv_both","kv_connector_module_path":"kvbm.vllm_integration.connector"}'

GPU KV pool in this configuration: 44,832 tokens.

Probe (temperature 0, thinking disabled, streaming):

1. Send document A (~30K tokens, "[doc 1000] … Question: what is the first code word?"). Answer: yankee (correct). TTFT 9.0 s.
2. Send four different ~30K-token documents E1–E4 to evict A from the GPU pool and from the 8 GB host tier (last one, E4, answers delta).
3. Send document A again.

Full probe script (calibrates tokens/word, checks the expected word, diffs kvbm_* counters): https://github.com/ryangu00/dell-pro-max-gb10-kvbm-ssd-kv-cache/blob/main/scripts/kvbm_probe.py

Expected Behavior

Either A is served from disk with the correct KV (answer yankee), or KVBM reports a miss / raises and vLLM recomputes the prefill.

Actual Behavior

Step 3 returns in 0.22 s with the answer delta (E4's answer). Metrics delta between step 1 and step 3: kvbm_onboard_blocks_d2d +1884 (= 30,156 tokens / 16), kvbm_offload_blocks_d2h +7536, kvbm_offload_blocks_h2d +7536. Engine log shows the createXferReq failure above at the moment of the "hit".

Controls on the same machine and model:

• No KVBM: step 3 TTFT 9.13 s, answer yankee (recompute, correct).
• KVBM with host tier only (no DYN_KVBM_DISK_CACHE_GB), evict 1: step 3 TTFT 0.25 s, answer yankee, kvbm_onboard_blocks_h2d +971 — host→device onboarding is correct.
• Setting DYN_KVBM_NIXL_BACKEND_POSIX=true / DYN_KVBM_NIXL_BACKEND_GDS_MT=false / DYN_KVBM_NIXL_BACKEND_GDS=false, and separately a cuFile force_compat_mode config via CUFILE_ENV_PATH_JSON, reproduce the identical failure.

Raw probe JSON and engine logs for every run: https://github.com/ryangu00/dell-pro-max-gb10-kvbm-ssd-kv-cache/tree/main/results

Environment

• Dynamo / KVBM: kvbm==1.4.2 (PyPI aarch64 wheel), used standalone via vllm serve --kv-transfer-config
• vLLM 0.27.1, torch 2.13.0+cu130, NIXL 1.3.1 (image's own), Python 3.12
• Driver 580.142, CUDA 13; no nvidia-fs module
• Hardware: single GB10 (Dell Pro Max with GB10), 128 GB unified memory, sm_121, aarch64, NVMe ext4 for the disk cache
• Model: Qwen/Qwen3-8B (dense; hybrid models fail earlier with Failed to promote local KV cache specs to one unified type, tracked separately in #8051)

Suggested behavior

1. If GDS_MT file registration fails at startup, either fail the disk tier loudly or fall back to the host-bounce (POSIX) path for disk↔device.
2. A failed transfer_blocks must mark the request's blocks as not loaded (or fail the request) instead of leaving the scheduler with stale physical blocks.