OutOfMemoryError

I have OOM error during inference but not during training.

This happens even with batch size of 1 and even with increasing the GPU memory.

torch.cuda.OutOfMemoryError: CUDA out of memory. Tried to allocate 11.25 GiB (GPU 0; 44.42 GiB total capacity; 36.96 GiB already allocated; 3.95 GiB free; 38.83 GiB reserved in total by PyTorch) If reserved memory is >> allocated memory try setting max_split_size_mb to avoid fragmentation. See documentation for Memory Management and PYTORCH_CUDA_ALLOC_CONF

I think its not OOM issue but something wrong with the Trainer increasing reserving memory.