Validation retains InfiniteDataLoader workers and prefetched batches between epochs

GPU comparison — 2026-09-06

Fresh measurements on PR #26014 confirm that the current finite/persistent design drains validation queues. The 1920px small-data check retained about 64 MiB less shared memory with identical predictions. Matched 20-epoch training took 4.8% longer on COCO8 and 11.7% longer on COCO128, with identical per-epoch metrics. Recommend no default loader change from these results alone; the original GB10 memory-pressure workload remains unverified.

Current status — 2026-09-05

• Lower validation prefetch shipped in #26057 (8.4.139): current build_dataloader uses 2 for unshuffled loaders, retaining 4 for shuffled training. The original report below describes 8.4.120 and prefetch 4; its 80-batch/~27-GiB queue estimate is historical (at depth 2, the same raw-image estimate is 40 batches/~13.2 GiB, before overhead).
• The reported GB10/1920px/batch16 memory-pressure case is not yet reproduced on current main. This session's real-image Linux/macOS worker check confirms workers remain alive after a pass and terminate at close(), with prefetch 2. It does not reproduce the original 95–100-GB pressure.
• #26014 remains open. Its current proposal is finite validation iteration with persistent workers, not unconditional worker teardown. Validate its current head against the current prefetch-2 baseline on the reported GB10 workload, measuring peak/settled PSS, epoch wall time, and unchanged metrics together.

Preserve the explicit maintainer decision to reuse validation workers during training. Historical matched measurements of the earlier #26014 teardown version showed unconditional teardown/recreation increased wall time ~25.3% and reduced throughput ~20.2%; recommend no action on that rejected approach. Later #26014 historical GPU trials were workload-dependent (COCO8 ~5.7% faster, COCO128 ~3.6% slower), not a universal speedup. Reducing workers and/or image/batch size is a current memory-pressure workaround. The remaining issue is bounded resource use under the reported configuration, not simply that a persistent worker is alive.

Original report and historical context

Search before asking

• I have searched the Ultralytics YOLO issues and found no similar bug report.

Ultralytics YOLO Component

Train, Val

Bug

General Summary

During detection training, validation uses an InfiniteDataLoader. Its worker processes and prefetched batches remain alive after a validation pass completes.

Detection validation also requests twice the configured worker count. With large images, this can retain tens of gigabytes of host memory, saturate the CPU with memory reclaim/compaction, and leave the GPU idle between epochs. There is no exception or traceback. The failure mode is excessive resource retention.

Actual Behavior

For the following training configuration:

• workers=15
• batch=16
• imgsz=1920
• cache=False
• 20 host CPUs
• NVIDIA GB10 with unified memory I observed:
• 15 training pt_data_worker processes
• 20 validation workers, because validation requests workers * 2, capped by the host CPU count
• Approximately 95–100 GB total memory usage
• 100% CPU utilization after validation completed
• kcompactd0 consuming a full CPU core
• GPU mostly idle during the resulting memory pressure
• Validation worker processes still alive after validation exited

Memory Breakdown

The training parent process had approximately:

• 38.6 GiB PSS
• Approximately 36.8 GiB shared-memory PSS
• Approximately 29.8 GiB NVIDIA allocation Representative training workers consumed approximately 1.7 GiB PSS each. This occurred with cache=False, so the memory usage was not caused by dataset caching.

Validation Prexetch Memory

At a validation batch size of 32, a raw uint8 image batch at 1920×1920 is approximately 337.5 MiB per batch. With 20 workers and a prefetch factor of 4, validation may queue as many as 20 workers × 4 batches/worker = 80 batches.

This corresponds to approximately 80 × 337.5 MiB ≈ 27 GiB of raw image data alone, before accounting for multiprocessing, shared-memory, tensor, and Python overhead. The combination of persistent validation workers and aggressive prefetching therefore creates substantial retained memory pressure even after the validation pass has completed.

Expected Behavior

• Training may continue using InfiniteDataLoader and persistent workers.
• Validation should use a normal finite torch.utils.data.DataLoader.
• Validation workers and prefetch queues should terminate after validation.
• workers=N should not unexpectedly create 2N validation workers.
• Validation should use a conservative prefetch factor to avoid large queued batches when training with high-resolution images.

Environment

 Ultralytics 8.4.120
 Python 3.12.13
 PyTorch 2.13.0+cu130
 Torchvision 0.28.0
 CUDA 13.0

 OS          Linux 6.17.0-1029-nvidia, aarch64
 CPU count   20
 RAM         121.7 GB
 GPU         NVIDIA GB10, 124610 MiB
 GPU count   1

 numpy                 1.26.4
 opencv-python         4.11.0.86
 pillow                12.3.0
 psutil                7.2.2
 polars                1.43.2
 ultralytics-thop      2.1.6
 ultralytics-platform  0.1.5

### Minimal Reproducible Example

 ```python
 from ultralytics.data.build import build_dataloader

 loader = build_dataloader(
     range(64),
     batch=8,
     workers=2,
     shuffle=False,
     device="cpu",
 )

 workers = list(loader.iterator._workers)

 try:
     print("loader type:", type(loader).__name__)
     list(loader)

     alive = [worker.is_alive() for worker in workers]
     print("workers alive after full iteration:", alive)
     print("worker PIDs:", [worker.pid for worker in workers])

     assert not any(alive), "Workers remain alive after the finite pass"
 finally:
     loader.close()

Additional

No response

Are you willing to submit a PR?

• Yes I'd like to help by submitting a PR!