容器内推理时 GPU 推理报错 MMU Fault: ENGINE GRAPHICS GPC2 GPCCLIENT_T1_0， fallback 到 CPU，重启容器解决

root@5d13d5c8e216:/app# pip list | grep paddle
paddleocr                      3.7.0
paddlepaddle-gpu               3.3.1
paddlex                        3.7.2

root@5d13d5c8e216:/app# paddle --version
<stdin>:3: DeprecationWarning: distutils Version classes are deprecated. Use packaging.version instead.
PaddlePaddle 3.3.1, compiled with
    with_avx: ON
    with_gpu: ON
    with_mkl: ON
    with_mkldnn: ON
    with_python: ON

root@fgg:/data/docparser# docker compose -f compose.yml exec -it ocr-worker-2 bash
root@5d13d5c8e216:/app# nvidia-smi 
Failed to initialize NVML: Unknown Error

| terminate called after throwing an instance of 'common::enforce::EnforceNotMet'
|   what():  (External) CUDA error(700), an illegal memory access was encountered. 
|   [Hint: 'cudaErrorIllegalAddress'. The device encountered a load or store instruction on an invalid memory address. This leaves the process in an inconsistentstate and any further CUDA work will return the same error. To continue using CUDA, the process must be terminated and relaunched. ] (at ../paddle/phi/core/platform/device/gpu/gpu_info.cc:348)
| 
| 
| 
| --------------------------------------
| C++ Traceback (most recent call last):
| --------------------------------------
| 0   paddle_infer::Predictor::Predictor(paddle::AnalysisConfig const&)
| 1   paddle::AnalysisPredictor::~AnalysisPredictor()
| 2   paddle::AnalysisPredictor::~AnalysisPredictor()
| 3   paddle::memory::allocation::StreamSafeCUDAAllocator::ReleaseImpl(phi::Place const&)
| 4   paddle::memory::allocation::AutoGrowthBestFitAllocator::FreeIdleChunks()
| 5   paddle::memory::allocation::CUDAAllocator::FreeImpl(phi::Allocation*)
| 
| ----------------------
| Error Message Summary:
| ----------------------
| FatalError: `Process abort signal` is detected by the operating system.
|   [TimeInfo: *** Aborted at 1790204916 (unix time) try "date -d @1790204916" if you are using GNU date ***]
|   [SignalInfo: *** SIGABRT (@0xa861) received by PID 43105 (TID 0x7f85478fd1c0) from PID 43105 ***]

# dmesg | grep -iE "xid|nvidia|fallen off the bus" | tail -30
[Tue Sep 22 15:52:05 2026] NVRM: Xid (PCI:0000:00:08): 31, pid=21673, name=celery, channel 0x00000098, intr 00000000. MMU Fault: ENGINE GRAPHICS GPC2 GPCCLIENT_T1_0 faulted @ 0x5603_dc0e9000. Fault is of type FAULT_PDE ACCESS_TYPE_VIRT_READ
[Tue Sep 22 15:52:06 2026] NVRM: Xid (PCI:0000:00:08): 31, pid=21674, name=celery, channel 0x000000a8, intr 00000000. MMU Fault: ENGINE GRAPHICS GPC2 GPCCLIENT_T1_0 faulted @ 0x55b9_669cc000. Fault is of type FAULT_PDE ACCESS_TYPE_VIRT_READ
[Tue Sep 22 15:57:48 2026] NVRM: Xid (PCI:0000:00:08): 31, pid=77848, name=celery, channel 0x0000009a, intr 00000000. MMU Fault: ENGINE GRAPHICS GPC2 GPCCLIENT_T1_0 faulted @ 0x5603_dd017000. Fault is of type FAULT_PDE ACCESS_TYPE_VIRT_READ
[Tue Sep 22 15:57:48 2026] NVRM: Xid (PCI:0000:00:08): 31, pid=68479, name=celery, channel 0x00000098, intr 00000000. MMU Fault: ENGINE GRAPHICS GPC2 GPCCLIENT_T1_0 faulted @ 0x55b9_66774000. Fault is of type FAULT_PDE ACCESS_TYPE_VIRT_READ
[Tue Sep 22 16:15:18 2026] NVRM: Xid (PCI:0000:00:08): 31, pid=902727, name=celery, channel 0x00000089, intr 00000000. MMU Fault: ENGINE GRAPHICS GPC2 GPCCLIENT_T1_0 faulted @ 0x55ab_ce92e000. Fault is of type FAULT_PDE ACCESS_TYPE_VIRT_READ
[Tue Sep 22 16:15:19 2026] NVRM: Xid (PCI:0000:00:08): 31, pid=902615, name=celery, channel 0x00000088, intr 00000000. MMU Fault: ENGINE GRAPHICS GPC2 GPCCLIENT_T1_0 faulted @ 0x55ab_ce8f5000. Fault is of type FAULT_PDE ACCESS_TYPE_VIRT_READ
[Wed Sep 23 17:05:12 2026] NVRM: Xid (PCI:0000:00:08): 31, pid=845434, name=celery, channel 0x00000078, intr 00000000. MMU Fault: ENGINE GRAPHICS GPC2 GPCCLIENT_T1_0 faulted @ 0x5615_65f02000. Fault is of type FAULT_PDE ACCESS_TYPE_VIRT_READ
[Wed Sep 23 17:21:52 2026] NVRM: Xid (PCI:0000:00:08): 31, pid=1021729, name=celery, channel 0x00000078, intr 00000000. MMU Fault: ENGINE GRAPHICS GPC2 GPCCLIENT_T1_0 faulted @ 0x55c7_ca5e1000. Fault is of type FAULT_PDE ACCESS_TYPE_VIRT_READ
[Wed Sep 23 19:05:30 2026] NVRM: Xid (PCI:0000:00:08): 31, pid=1372089, name=celery, channel 0x00000068, intr 00000000. MMU Fault: ENGINE GRAPHICS GPC2 GPCCLIENT_T1_0 faulted @ 0x55f1_ad9c0000. Fault is of type FAULT_PDE ACCESS_TYPE_VIRT_READ
[Thu Sep 24 06:59:52 2026] NVRM: Xid (PCI:0000:00:08): 31, pid=2596962, name=celery, channel 0x00000077, intr 00000000. MMU Fault: ENGINE GRAPHICS GPC2 GPCCLIENT_T1_0 faulted @ 0x563e_fc07f000. Fault is of type FAULT_PDE ACCESS_TYPE_VIRT_READ
[Thu Sep 24 07:00:47 2026] NVRM: Xid (PCI:0000:00:08): 31, pid=2685061, name=celery, channel 0x00000077, intr 00000000. MMU Fault: ENGINE GRAPHICS GPC2 GPCCLIENT_T1_0 faulted @ 0x563e_fb94a000. Fault is of type FAULT_PDE ACCESS_TYPE_VIRT_READ
[Thu Sep 24 07:07:23 2026] NVRM: Xid (PCI:0000:00:08): 31, pid=2693218, name=celery, channel 0x00000077, intr 00000000. MMU Fault: ENGINE GRAPHICS GPC2 GPCCLIENT_T1_0 faulted @ 0x563e_fc44c000. Fault is of type FAULT_PDE ACCESS_TYPE_VIRT_READ

证据链（谁先谁后）

  ┌───────┬─────────────────────────────────────────────────────────────────────┬────────────────────────────────────────────────────────────────────────────────────────────────────┐
  │ 时刻  │                                事件                                 │                                              证据来源                                              │
  ├───────┼─────────────────────────────────────────────────────────────────────┼────────────────────────────────────────────────────────────────────────────────────────────────────┤
  │ T     │ 本进程提交的 kernel 读了非法地址 → GPU MMU Fault                    │ dmesg：Xid 31, pid=845434, name=celery … faulted @ 0x5615_65f02000，pid 就是后来落 core 的那个进程 │
  ├───────┼─────────────────────────────────────────────────────────────────────┼────────────────────────────────────────────────────────────────────────────────────────────────────┤
  │ T     │ 该地址落在这个进程自己的堆段内（0x56155cbcd000–0x561568b44000）     │ core A 的 PT_LOAD 段                                                                               │
  ├───────┼─────────────────────────────────────────────────────────────────────┼────────────────────────────────────────────────────────────────────────────────────────────────────┤
  │ T+72s │ CUDA runtime 把 context 标坏后，下一次 CUDA 调用返回 error 700      │ core 里 ExternalError: CUDA error(700) … gpu_info.cc:348                                           │
  ├───────┼─────────────────────────────────────────────────────────────────────┼────────────────────────────────────────────────────────────────────────────────────────────────────┤
  │ T+72s │ Paddle 抛 EnforceNotMet → 析构里二次抛出 → std::terminate → abort() │ core B 里的两条 C++ 栈 + 你贴的 terminate called …                                                 │
  ├───────┼─────────────────────────────────────────────────────────────────────┼────────────────────────────────────────────────────────────────────────────────────────────────────┤
  │ T+72s │ SIGABRT → glog banner → core                                        │ Aborted at 1790154384 / 1790204519                                                                 │
  └───────┴─────────────────────────────────────────────────────────────────────┴────────────────────────────────────────────────────────────────────────────────────────────────────┘

  关键在于 dmesg 里的 pid 是发起那个 GPU channel 的进程，也就是谁提交的 kernel 谁背这个 fault。两次 core 的 dmesg pid 与 core 文件名 pid
  完全一致（845434、2685061），所以不是"别人先崩、本进程被动躺枪"。再加上故障地址落在它自己的 CPU 堆里（UVA 下设备 VA 不可能和 brk 堆重叠），指针就是从它自己内存里来的 → 是它自己把主机指针喂给了 kernel。

  层次关系（700 是"回声"，不是新故障）

  本进程提交 kernel，kernel 读到无效地址
     ↓        ← 这是唯一的原始事件，发生在 GPU 侧
  驱动记 Xid 31（dmesg），并把该 context 标记为损坏
     ↓
  CUDA runtime 在进程下次触到该 context 时返回 cudaErrorIllegalAddress(=700)
     ↓
  Paddle 把它包成 EnforceNotMet；此时若发生在析构里且正在 unwinding → terminate → abort

  cudaErrorIllegalAddress 的字面意思就是"设备遇到了非法内存访问"，它就是那次 MMU fault 的运行时表达，不是第二个 bug。所以严格说：700 和 abort 都只是信使，原始缺陷在"把坏指针送进
  kernel"的那段推理/加载代码里（Paddle/PaddleX 侧）。