GPU memory leak in _fast_paste_back causes CUBLAS_STATUS_NOT_INITIALIZED crash on long videos

Author: caiyilianCreated Jun 26, 2026Updated Jul 10, 2026

Description

When processing long videos (>500 frames) with --execution-provider cuda, Deep-Live-Cam crashes partway through with:

  • CUBLAS_STATUS_NOT_INITIALIZED
  • CUDA failure 715: illegal instruction

The crash is caused by a GPU memory leak in _fast_paste_back in modules/processors/frame/face_swapper.py.

Root Cause

_fast_paste_back has a PyTorch CUDA path that creates intermediate tensors on the GPU but never explicitly frees them:

mask_t = torch.from_numpy(mask_crop).float().cuda()
fake_t = torch.from_numpy(bgr_fake_crop).float().cuda()
tgt_t = torch.from_numpy(target_crop).float().cuda()
blended = (mask_t * fake_t + (1.0 - mask_t) * tgt_t).to(torch.uint8).cpu().numpy()

mask_t, fake_t, tgt_t, and blended persist in GPU memory until Python's GC runs. ONNX Runtime (insightface) also holds GPU memory on the same device. Over hundreds of frames, the unreleased PyTorch tensors accumulate, exhausting VRAM and corrupting the CUDA context.

Reproduction

  1. Select a source face image and a target video with ~1000+ frames at 1080p
  2. Run: python run.py -s face.jpg -t test.mp4 -o output.mp4 --execution-provider cuda
  3. Observe crash partway through (typically after ~300-500 frames on a 6GB GPU)

Environment

  • GPU: NVIDIA RTX 2060 6GB
  • Driver: 596.21
  • CUDA: 12.4
  • OS: Windows 11
  • Python: 3.11.14
  • onnxruntime-gpu: 1.18.0
  • PyTorch: 2.6.0+cu124

Fix

  1. Add del mask_t, fake_t, tgt_t, blended after the blend operation to immediately release tensor references
  2. Add periodic gc.collect() + torch.cuda.empty_cache() every ~50 frames in _run_pipe_pipeline
  3. Make release_resources() attempt torch.cuda.empty_cache() via runtime import instead of requiring a module-level HAS_TORCH flag

I have a verified working fix (benchmarked: 1129 frames completed without crash, ~8.9 fps) and can open a PR.