#26109 Random SIGSEGV (exit code 139) and occasional complete system freeze during YOLO training

Search before asking

I have searched the Ultralytics YOLO issues and discussions and found no similar questions.

Question

Random SIGSEGV (exit code 139), CUDA errors, and occasional complete system freeze during YOLOv8 OBB training on RTX 5090

Summary

I am experiencing nondeterministic native crashes while training a YOLOv8 OBB model on an NVIDIA GeForce RTX 5090 under Linux.

The training process sometimes terminates with:

Process finished with exit code 139 (interrupted by signal 11: SIGSEGV)

I have also observed a more severe failure mode in which the entire graphical desktop becomes unresponsive, SSH connections to the machine stop working, and the training is interrupted. A reboot may be required to recover the system.

The crash does not always occur at the same epoch or batch.

In the latest diagnostic run, the process did not immediately terminate with SIGSEGV. Instead, it encountered several CUDA out-of-memory errors, automatically reduced the batch size multiple times, and finally failed with:

torch.AcceleratorError: CUDA error: unknown error

Systemd has separately recorded multiple previous Python processes that terminated with SIGSEGV.

Expected Behavior

The YOLO training should complete normally.

If GPU memory is insufficient, I would expect a recoverable torch.cuda.OutOfMemoryError or another Python exception with a useful traceback. The Python process should not terminate with SIGSEGV, and the entire desktop and SSH service should not become unresponsive.

Actual Behavior

I have observed two failure modes.

Failure mode 1: Native Python process crash

The process exits without a normal Python traceback:

Process finished with exit code 139 (interrupted by signal 11: SIGSEGV)

The epoch at which this occurs is not consistent between runs.

Failure mode 2: Complete system freeze

In some runs:

The graphical desktop becomes completely unresponsive.
SSH connections to the machine stop working.
The YOLO training stops.
The machine may require a forced reboot.

Reproduction

The training uses a local checkout of Ultralytics.

The run resumes from a YOLO checkpoint. The equivalent public API operation is:

from ultralytics import YOLO

model = YOLO("runs/obb/train-9/weights/last.pt")
model.train(resume=True)

The complete configuration restored from the checkpoint is summarized below.

The resume operation is recognized correctly:

Resuming training runs/obb/train-9/weights/last.pt from epoch 147 to 150 total epochs
Closing dataloader mosaic

The failure is nondeterministic and may require sustained training to reproduce.

Training Configuration

Task: Oriented Bounding Box detection
Model: YOLOv8l-obb
Model parameters: 44,484,072
Trainable gradients: 44,484,056
Number of classes: 7
Target epochs: 150
Resume epoch: 147 of 150
Image size: 1024
Initial batch size: 16
DataLoader workers: 2
Device: CUDA device 0
AMP: enabled
torch.compile: disabled
Dataset cache: disabled
Deterministic mode: enabled
Optimizer setting: auto
Optimizer selected by Ultralytics: MuSGD
Learning rate: 0.01
Momentum: 0.9
Weight decay: 0.0005
Mosaic probability: 0.7
MixUp probability: 0.05
Rotation augmentation: 15 degrees
Scale augmentation: 0.2
Translation augmentation: 0.1
Final mosaic-closing period: 20 epochs
Training images: 82,091
Validation images: 8,482
Corrupt training images detected: 0
Corrupt validation images detected: 0

Relevant configuration output:

amp=True
batch=16
cache=False
channels_last=False
close_mosaic=20
compile=False
data=<DATASET_ROOT>/data.yaml
degrees=15
deterministic=True
device=0
epochs=150
imgsz=1024
mixup=0.05
mosaic=0.7
optimizer=auto
resume=runs/obb/train-9/weights/last.pt
task=obb
val=True
workers=2

Environment

Hardware and operating system

OS: Ubuntu 22.04.5 LTS, x86_64
Kernel: Linux 6.8.0-138-generic
CPU: Intel Core i9-14900KF
CPU topology: 24 physical cores / 32 logical CPUs
GPU: NVIDIA GeForce RTX 5090
GPU memory: 32,102 MiB
GPU compute capability: 12.0, Blackwell
NVIDIA driver: 595.84
NVIDIA kernel module: NVIDIA Open Kernel Module
System RAM: approximately 64 GB

Python and machine-learning environment

Python: 3.10.21
Ultralytics: 8.4.124
Ultralytics installation: local source checkout
PyTorch: 2.12.1+cu130
torchvision: 0.27.1+cu130
Triton: 3.7.1
NumPy: 2.2.6
CUDA used to build PyTorch: 13.0
System CUDA runtime: 13.2.51
Active cuDNN runtime: 9.20.0
torch.backends.cudnn.version(): 92000

Relevant PyTorch environment information:

PyTorch version: 2.12.1+cu130
Is debug build: False
CUDA used to build PyTorch: 13.0
ROCM used to build PyTorch: N/A

OS: Ubuntu 22.04.5 LTS (x86_64)
GCC version: Ubuntu 11.4.0
Libc version: glibc-2.35

Python version: 3.10.21
Python platform: Linux-6.8.0-138-generic-x86_64-with-glibc2.35
Is CUDA available: True
CUDA runtime version: 13.2.51
GPU: NVIDIA GeForce RTX 5090
NVIDIA driver version: 595.84

Relevant installed packages:

numpy==2.2.6
nvidia-cublas==13.1.1.3
nvidia-cuda-cupti==13.0.85
nvidia-cuda-nvrtc==13.0.88
nvidia-cuda-runtime==13.0.96
nvidia-cudnn-cu13==9.20.0.48
nvidia-cufft==12.0.0.61
nvidia-curand==10.4.0.35
nvidia-cusolver==12.0.4.66
nvidia-cusparse==12.6.3.3
nvidia-cusparselt-cu13==0.8.1
nvidia-nccl-cu13==2.29.7
nvidia-nvjitlink==13.0.88
nvidia-nvtx==13.0.85
torch==2.12.1+cu130
torchvision==0.27.1+cu130
triton==3.7.1
ultralytics==8.4.124

Loaded CUDA Libraries

I executed a CUDA convolution and backward pass, synchronized the GPU, and inspected /proc/self/maps.

PyTorch loaded cuDNN and cuBLAS from the active Conda environment:

$CONDA_PREFIX/lib/python3.10/site-packages/nvidia/cu13/lib/libcublas.so.13
$CONDA_PREFIX/lib/python3.10/site-packages/nvidia/cu13/lib/libcublasLt.so.13
$CONDA_PREFIX/lib/python3.10/site-packages/nvidia/cudnn/lib/libcudnn.so.9
$CONDA_PREFIX/lib/python3.10/site-packages/nvidia/cudnn/lib/libcudnn_engines_precompiled.so.9
$CONDA_PREFIX/lib/python3.10/site-packages/nvidia/cudnn/lib/libcudnn_engines_runtime_compiled.so.9
$CONDA_PREFIX/lib/python3.10/site-packages/nvidia/cudnn/lib/libcudnn_graph.so.9
$CONDA_PREFIX/lib/python3.10/site-packages/nvidia/cudnn/lib/libcudnn_heuristic.so.9

The NVIDIA CUDA driver library was loaded from:

/usr/lib/x86_64-linux-gnu/libcuda.so.595.84

System cuDNN 9.24 libraries also exist under /usr/lib, but they were not loaded by this PyTorch process. Therefore, there is currently no evidence that this process mixed the system cuDNN libraries with the PyTorch-provided cuDNN libraries.

System Resource Observations

A top snapshot taken during training showed approximately:

Physical memory usage: 19.2% of 64 GB
Swap usage: 1.4%
CPU idle: 92.2%
I/O wait: 0.1%

The main processes used approximately:

Main Python process: 4.2 GB resident memory
DataLoader worker 1: 3.1 GB resident memory
DataLoader worker 2: 2.8 GB resident memory

There was no obvious host RAM, CPU, or disk I/O exhaustion at the time of the snapshot.

This does not include GPU memory usage.

CUDA OOM and Automatic Batch Reduction

The resumed training initially started with batch size 16.

Ultralytics then automatically reduced the batch size:

WARNING: CUDA out of memory with batch=16.
Reducing to batch=8 and retrying (1/3).

The retry with batch size 8 also failed:

WARNING: CUDA out of memory with batch=8.
Reducing to batch=4 and retrying (2/3).

At batch size 4, the TaskAlignedAssigner encountered another CUDA OOM and used its CPU fallback:

WARNING: CUDA OutOfMemoryError in TaskAlignedAssigner, using CPU

Ultralytics then reduced the batch size again:

WARNING: CUDA out of memory with batch=4.
Reducing to batch=2 and retrying (3/3).

The model, DataLoaders, optimizer, and scheduler were rebuilt after every batch-size reduction.

After the final retry, the process failed with:

torch.AcceleratorError: CUDA error: unknown error

Final CUDA Exception

The Python traceback ended at:

File "ultralytics/utils/tal.py", line 168, in get_pos_mask
    mask_pos = mask_topk * mask_in_gts * mask_gt

torch.AcceleratorError: CUDA error: unknown error

The complete error message was:

torch.AcceleratorError: CUDA error: unknown error
Search for `cudaErrorUnknown` in the CUDA Runtime API documentation.

CUDA kernel errors might be asynchronously reported at some other API call,
so the stack trace below might be incorrect.

For debugging consider passing CUDA_LAUNCH_BLOCKING=1.
Compile with TORCH_USE_CUDA_DSA to enable device-side assertions.

The captured C++ traceback included:

at::native::gpu_kernel_impl
at::native::mul_kernel_cuda
BinaryMulKernel.compute_120.cudafe1.cpp

The compute_120 filename is consistent with the RTX 5090 Blackwell compute capability.

Because CUDA operations are asynchronous, the tensor multiplication in tal.py may only be where the error was detected, rather than the original source of the GPU failure.

Secondary DataLoader Error

After the main CUDA exception, the pinned-memory thread failed with:

ConnectionResetError: [Errno 104] Connection reset by peer

The traceback included:

torch/utils/data/_utils/pin_memory.py
torch/multiprocessing/reductions.py
multiprocessing/resource_sharer.py
multiprocessing/connection.py

This appears more likely to be a secondary consequence of the main process failing and closing the DataLoader worker connection, rather than the original cause.

Recorded SIGSEGV Core Dumps

coredumpctl shows that multiple Python processes have terminated with SIGSEGV:

Tue 2026-09-08 00:54:16 CST
PID: 3289
Signal: SIGSEGV
Core file: truncated
Core size: 516.1 MB
Executable: $CONDA_PREFIX/bin/python3.10

Fri 2026-09-04 16:34:21 CST
PID: 4150
Signal: SIGSEGV
Core file: missing
Executable: $CONDA_PREFIX/bin/python3.10

Thu 2026-09-03 16:38:32 CST
PID: 3424
Signal: SIGSEGV
Core file: missing
Executable: $CONDA_PREFIX/bin/python3.10

This confirms that the previously observed exit-code-139 failures were native SIGSEGV crashes rather than normal Python exceptions.

A native backtrace from PID 3289 has not yet been successfully extracted. Its core dump is available but marked as truncated.

Kernel Log Observations

I checked the kernel log using:

sudo dmesg -T |
grep -Ei "oom|out of memory|killed process|nvrm|xid|segfault"

No OOM-killer event, NVIDIA GPU Xid error, or kernel-reported segmentation fault related to the training failure was found.

The only matching lines were similar to:

r8169 ... eth0: RTL8125B ... XID 641
NVRM: loading NVIDIA UNIX Open Kernel Module for x86_64 595.84

The r8169 line belongs to the Realtek Ethernet adapter and is not an NVIDIA GPU Xid error.

The NVRM line only reports that the NVIDIA kernel module was loaded.

Dataset Warnings

Ultralytics reported duplicate labels in 10 training images and removed one duplicate label from each affected image.

No corrupt images were detected:

Training set: 82,091 images, 0 backgrounds, 0 corrupt
Validation set: 8,482 images, 0 backgrounds, 0 corrupt

The duplicate labels should be cleaned, but they do not currently appear to explain the native SIGSEGV, CUDA driver error, or complete system freeze.

Potentially Relevant Findings

1. Severe GPU memory pressure

The latest run could not complete its first resumed batch with batch sizes 16, 8, or 4.

The repeated CUDA OOM errors indicate severe GPU memory pressure at:

YOLOv8l-obb
imgsz=1024
batch=16
RTX 5090 with 32 GB VRAM

2. CUDA context or driver error after OOM recovery

After multiple OOM recovery attempts and DataLoader/optimizer rebuilds, the CUDA runtime returned:

CUDA error: unknown error

This may indicate that the CUDA context or NVIDIA driver entered an invalid error state after the repeated allocation failures. This has not yet been conclusively confirmed.

3. Multiple independent SIGSEGV crashes

Systemd has recorded at least three separate SIGSEGV terminations of the same Python environment.

The crashes occurred on different dates, which suggests that this is not a single accidental termination.

4. Complete system-level freezes

In some runs, the failure affects more than the Python process:

The graphical desktop freezes.
SSH becomes unreachable.
Training stops.
A forced reboot may be required.

This behavior suggests a possible GPU driver, PCIe, firmware, power, or kernel-level failure, although no NVIDIA Xid has yet been found in the checked kernel log.

5. cuDNN library conflict appears unlikely

Although the system contains cuDNN 9.24 libraries under /usr/lib, runtime inspection showed that PyTorch loaded cuDNN 9.20 from the active Conda environment.

Therefore, a mixed cuDNN runtime does not currently appear to be the primary cause.

6. DataLoader error appears secondary

The pinned-memory ConnectionResetError happened after the main CUDA failure. It is probably a consequence of the failed training process rather than the original source of the problem.

Current Assessment

The strongest observed signal is severe GPU memory pressure followed by a CUDA driver/runtime failure.

The latest run encountered CUDA OOM with batch sizes 16, 8, and 4. After automatic recovery and reduction to batch size 2, CUDA returned cudaErrorUnknown.

Separately, systemd has recorded multiple native Python SIGSEGV crashes, and some training runs have caused the entire graphical desktop and SSH service to become unresponsive.

The machine uses:

RTX 5090
NVIDIA driver 595.84
PyTorch 2.12.1+cu130
CUDA 13.0/13.2

This environment is similar to previously reported nondeterministic CUDA and libcuda.so.1 failures on RTX 5090 systems.

A native GDB backtrace from the available core dump is still needed to determine whether the SIGSEGV occurred inside:

libcuda.so.1
libtorch_cuda.so
cuDNN
a PyTorch CUDA kernel
OpenCV or another image-decoding library
another native dependency

Related Reports

Potentially related upstream reports include:

PyTorch:
https://github.com/pytorch/pytorch/issues/182279
NVIDIA Developer Forums:
https://forums.developer.nvidia.com/t/segmentation-fault-on-rtx-5090-with-cuda-13-during-repeated-pytorch-cuda-forward-passes/368902
PyTorch/YOLO high-VRAM-pressure CUDA failure:
https://github.com/pytorch/pytorch/issues/178491

The first report describes nondeterministic SIGSEGV failures on an RTX 5090 with CUDA 13 and a 595-series NVIDIA driver. A GDB backtrace in that report points to libcuda.so.1.

My environment differs in the following ways:

Reported issue:
PyTorch 2.11.0+cu130
NVIDIA driver 595.71.05

My environment:
PyTorch 2.12.1+cu130
NVIDIA driver 595.84

Questions

Is this a known issue with RTX 5090, CUDA 13, and NVIDIA driver 595.84?
Can repeated CUDA OOM recovery attempts leave the CUDA context in a state that later produces cudaErrorUnknown?
Is the automatic batch-size reduction and trainer rebuild path expected to recover safely from repeated OOM errors?
Is there a recommended stable combination of NVIDIA driver, PyTorch, CUDA, and cuDNN for YOLO training on RTX 5090?
Could the complete desktop and SSH freeze indicate a GPU driver or firmware problem even when no NVIDIA Xid appears in the current dmesg output?
What additional diagnostics would be most useful?
Should I disable pinned memory or set workers=0 to rule out a DataLoader-related failure?
Would a complete GDB backtrace from the truncated core dump still be useful?

Additional Diagnostics in Progress

I am attempting to extract the available core dump using:

coredumpctl info 3289
sudo coredumpctl debug 3289

Inside GDB:

set pagination off
thread apply all bt

I am also planning to repeat the training with:

workers=0
cache=False
a smaller fixed batch size
CUDA_LAUNCH_BLOCKING=1
PYTHONFAULTHANDLER=1
TORCH_SHOW_CPP_STACKTRACES=1

I can provide the resulting logs and native backtrace when available.

Additional

No response