Intermittent libcuda.so segmentation fault (error 6) in make_motion_template (Driver 610.43.02)
Describe the bug
When running video processing, the pipeline randomly crashes with a segmentation fault inside libcuda.so. The crash always occurs during the make_motion_template call.
The issue is strictly intermittent and probabilistically tied to the number of frames processed. The failure rate is roughly ~0.3% per frame/call.
- Processing 78 frames yields an 83% success rate.
- Processing 250 frames drops the success rate to 33%.
This indicates an accumulated probability of failure over independent iterations, rather than a cumulative memory leak (verified ~13.7GB free VRAM at the time of crashes). The system logs show the driver received a null pointer and attempted to read from it (error 6):
python[15381]: segfault at 44 ip 0000763b77ac73c5 sp 00007ffca6945570 error 6 in libcuda.so.610.43.02[2c73c5,763b779a8000+127a000]The exact trace varies slightly between executions but always points to the linear layers inside the motion extractor: File "/app/src/utils/camera.py", line 68 in get_rotation_matrix File "/app/src/live_portrait_pipeline.py", line 56 in make_motion_template File "torch/nn/modules/linear.py", line 116 in forward File "/app/src/modules/motion_extractor.py", line 34 in forward File "/app/src/live_portrait_wrapper.py", line 138 in get_kp_info File "/app/src/live_portrait_pipeline.py", line 54 in make_motion_template
nvironment
GPU: NVIDIA GeForce RTX 4070 Ti SUPER (Ada, sm_89), 16376 MiB
Host Driver: 610.43.02
LivePortrait Commit: 9b294b3
PyTorch: Tested on 2.3.0+cu121, 2.5.1+cu121, and 2.5.1+cu118
ONNXRuntime: 1.18.0 (NumPy 1.26.4, Python 3.10.12)
Container: nvidia/cuda:12.1.1-cudnn8-devel-ubuntu22.04What we have already ruled out (Troubleshooting done)
To save time, we systematically tested and ruled out the common culprits:
ONNXRuntime vs. PyTorch CUDA context clash:
Switching ONNX to CPUExecutionProvider is not enough to test this, as onnxruntime-gpu loads its CUDA dynamic libraries on import. We completely uninstalled onnxruntime-gpu, replaced it with the pure CPU onnxruntime package, and the segfault still occurred.
VRAM Fragmentation / OOM:
There is over 13.7GB of free VRAM when the crash happens. Setting PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True had absolutely no effect.
Async Stream Pile-up / Thread-safety:
Adding torch.cuda.synchronize() doesn't mitigate it because the loop already implicitly synchronizes at every frame via a .cpu().numpy() call to copy the results. There is no async queue accumulating.
Container Isolation:
We tested 12 different environment/dependency variables inside the Docker container, and the failure rate remained completely static at ~33% for 250 frames. The variable causing the issue appears to be strictly related to the host's libcuda.so handling the context.Source: KlingAIResearch/LivePortrait