Xid 31 MMU Fault (FAULT_PDE, fixed per-box VA) on 610.57.04 under rapid Vulkan session churn; deterministic single-box repro, 580.159.03 clean

Title: Xid 31 MMU Fault (FAULT_PDE, fixed per-box VA) on driver 610.57.04 triggered by rapid GPU-context/session churn — deterministic single-machine repro; 580.159.03 unaffected

Summary

Linux driver 610.57.04 (open kernel modules) takes recurring Xid 31 MMU faults of type FAULT_PDE ACCESS_TYPE_VIRT_READ on the GRAPHICS engine when a Vulkan rendering process repeatedly creates and destroys GPU sessions/context resources. The faulting virtual address is near-constant per machine across processes and fault events, which points at a driver-internal page-table entry rather than an application-supplied pointer.

The identical workload, identical binaries, and identical kernel run clean on 580.159.03 (zero Xid 31 across 704 production nodes over the same window). Rolling back to 580.159.03 stopped the faults immediately in three separate environments. We have a deterministic single-machine reproduction that faults within 2–3 minutes.

We could not find this signature in the 610.57.04 release notes or in this tracker (closest reports involve CUDA JIT cache growth or desktop/browser workloads on GeForce, different triggers).

Impact

• Fleet-wide rollout of 610.57.04 aborted and rolled back from production; every GPU node is pinned to 580.159.03 until a driver passes our (now extended) qualification gate.
• On affected nodes: fault → render thread stalls 20–100 s → container liveness kills the process → restart loop approximately every 5 minutes.
• Because the trigger is session churn, any cloud/streaming product that cycles user sessions on a GPU is exposed.

Environment (minimal repro)

Also observed in production on T4 (g4dn), A10G (g5), and L40S (g6e).

Fault signature

Production (L40S/T4 nodes):

NVRM: Xid (PCI:0000:34:00): 31, pid=21311, name=Renny-Linux-Tes, channel 0x000000a8,
MMU Fault: ENGINE GRAPHICS GPC9 GPCCLIENT_T1_4 faulted @ 0x100_1ec00000.
Fault is of type FAULT_PDE ACCESS_TYPE_VIRT_READ

Minimal single-machine repro (A10G):

NVRM: Xid (PCI:0000:00:1e): 31, pid=84945, name=Renny-Linux-Tes, channel 0x74,
MMU Fault: ENGINE GRAPHICS GPC0 GPCCLIENT_T1_1 faulted @ 0x100_14000000.
Fault is of type FAULT_PDE ACCESS_TYPE_VIRT_READ

Key properties:

• Fixed VA per box. Production nodes faulted repeatedly in the 0x100_1ec0_0000 .. 0x100_1ec5_0000 region; the bench A10G uses its own fixed 0x100_1400_0000. Same pattern, different constant, across machines. The VA is driver-internal: it is not visible from application logs or nvidia-smi.
• Deterministic across different PIDs and different processes on the same box.
• GPCCLIENT_T1_* / GRAPHICS engine in every occurrence.
• Accompanied by DCGM_FI_DEV_XID_ERRORS err_code=31 with DCGM_FI_DRIVER_VERSION=610.57.04 on every faulting node; zero Xid 31 on any 580.159.03 node in the same fleet/window.

Reproduction

Single machine, single application process, no clustering required:

1. Install 610.57.04 open modules on the kernel above; run a Vulkan application that performs a full GPU session bring-up/teardown cycle (initialize per-session GPU resources, render briefly, tear everything down).
2. Repeat the cycle approximately once per minute.
3. Within 2–3 minutes (≈2–3 cycles), Xid 31 with the signature above appears; the application's render thread then stalls 20–100 s (consistent with the faulting channel being poisoned).

Controls we ran on the same box:

Variables already ruled out (each by controlled single-variable change)

• Kernel: 6.17 vs 7.0 — both fault on 610; 580 and 610 nodes ran the same 6.17.0-1019-aws.
• Module flavour: open and proprietary both fault (proprietary faults less often but is not a fix; separately, proprietary cannot initialize Blackwell at all, so it is not an escape hatch).
• Container/Kubernetes stack: reproduces with a stock device plugin/toolkit and also on a plain single-node microk8s box; a real managed-EKS run with idle renderers was the clean negative control. Runtime (containerd/runc/kubelet/cgroup/CDI) is not implicated.
• Multi-tenant GPU sharing / time-slicing depth: faults with a single process on an exclusive GPU.
• Application version: multiple builds/images all fault on 610, all clean on 580.
• Install flags (nvidia-drm, libglvnd combinations): no effect.

Questions / asks

1. Does the fixed per-box faulting VA in the 0x100_1XX0_0000 range ring a bell internally — e.g. a persistent driver-side allocation whose page-table root is torn down with the channel/context but still referenced by an in-flight or late-arriving graphics access?
2. Was there a change in GPU VA/page-table lifetime handling between the 580 and 610 branches around session/channel teardown?
3. Can you attempt reproduction from the recipe above? We can provide full nvidia-bug-report.sh captures from a faulting box, dmesg logs, and DCGM records.
4. If a fixed driver exists or is planned (we note 615.71.09 is now released), please point us at it — we will re-run our qualification gate (which now mandates one-hour session-churn testing at production time-slice depth on every GPU generation in our fleet).

Happy to provide artifacts, run instrumented builds, or coordinate with a customer engineer on our side.