GPU memory leak in _fast_paste_back causes CUBLAS_STATUS_NOT_INITIALIZED crash on long videos
Author: caiyilianCreated Jun 26, 2026Updated Jul 10, 2026
Description
When processing long videos (>500 frames) with --execution-provider cuda, Deep-Live-Cam crashes partway through with:
CUBLAS_STATUS_NOT_INITIALIZEDCUDA failure 715: illegal instruction
The crash is caused by a GPU memory leak in _fast_paste_back in modules/processors/frame/face_swapper.py.
Root Cause
_fast_paste_back has a PyTorch CUDA path that creates intermediate tensors on the GPU but never explicitly frees them:
mask_t = torch.from_numpy(mask_crop).float().cuda()
fake_t = torch.from_numpy(bgr_fake_crop).float().cuda()
tgt_t = torch.from_numpy(target_crop).float().cuda()
blended = (mask_t * fake_t + (1.0 - mask_t) * tgt_t).to(torch.uint8).cpu().numpy()
mask_t, fake_t, tgt_t, and blended persist in GPU memory until Python's GC runs. ONNX Runtime (insightface) also holds GPU memory on the same device. Over hundreds of frames, the unreleased PyTorch tensors accumulate, exhausting VRAM and corrupting the CUDA context.
Reproduction
- Select a source face image and a target video with ~1000+ frames at 1080p
- Run:
python run.py -s face.jpg -t test.mp4 -o output.mp4 --execution-provider cuda - Observe crash partway through (typically after ~300-500 frames on a 6GB GPU)
Environment
- GPU: NVIDIA RTX 2060 6GB
- Driver: 596.21
- CUDA: 12.4
- OS: Windows 11
- Python: 3.11.14
- onnxruntime-gpu: 1.18.0
- PyTorch: 2.6.0+cu124
Fix
- Add
del mask_t, fake_t, tgt_t, blendedafter the blend operation to immediately release tensor references - Add periodic
gc.collect()+torch.cuda.empty_cache()every ~50 frames in_run_pipe_pipeline - Make
release_resources()attempttorch.cuda.empty_cache()via runtime import instead of requiring a module-levelHAS_TORCHflag
I have a verified working fix (benchmarked: 1129 frames completed without crash, ~8.9 fps) and can open a PR.
Source: hacksider/Deep-Live-Cam