RTX 2060 Super - SAEHD training Memory Error even with Batch size 1
Author: StingraysevenCreated Jan 7, 2022Updated Oct 3, 2023
Tried lowering all kinds of resolutions and batch size to no avail. Currently at batch size 1 it's still giving me a memory error. I don't think this should happen on my GPU.
Windows 10.0.18363 Build 18363 AMD Ryzen 5 3600 Nvidia RTX 2060 Super
CMD
===================== Model Summary ======================
== ==
== Model name: new_SAEHD ==
== ==
== Current iteration: 0 ==
== ==
==------------------- Model Options --------------------==
== ==
== resolution: 64 ==
== face_type: wf ==
== models_opt_on_gpu: True ==
== archi: df ==
== ae_dims: 64 ==
== e_dims: 32 ==
== d_dims: 32 ==
== d_mask_dims: 22 ==
== masked_training: True ==
== eyes_mouth_prio: False ==
== uniform_yaw: True ==
== blur_out_mask: False ==
== adabelief: True ==
== lr_dropout: n ==
== random_warp: False ==
== random_hsv_power: 0.0 ==
== true_face_power: 0.0 ==
== face_style_power: 0.0 ==
== bg_style_power: 0.0 ==
== ct_mode: none ==
== clipgrad: True ==
== pretrain: True ==
== autobackup_hour: 1 ==
== write_preview_history: True ==
== target_iter: 0 ==
== random_src_flip: True ==
== random_dst_flip: True ==
== batch_size: 1 ==
== gan_power: 0.0 ==
== gan_patch_size: 16 ==
== gan_dims: 16 ==
== ==
==--------------------- Running On ---------------------==
== ==
== Device index: 0 ==
== Name: NVIDIA GeForce RTX 2060 SUPER ==
== VRAM: 6.17GB ==
== ==
==========================================================
Starting. Press "Enter" to stop training and save model.
Trying to do the first iteration. If an error occurs, reduce the model parameters.
!!!
Windows 10 users IMPORTANT notice. You should set this setting in order to work correctly.
https://i.imgur.com/B7cmDCB.jpg
!!!
Error:
Traceback (most recent call last):
File "D:\Program Files (x86)\Everything Deepfake\DeepFaceLab\DeepFaceLab_NVIDIA_up_to_RTX2080Ti\_internal\DeepFaceLab\mainscripts\Trainer.py", line 159, in trainerThread
model_save()
File "D:\Program Files (x86)\Everything Deepfake\DeepFaceLab\DeepFaceLab_NVIDIA_up_to_RTX2080Ti\_internal\DeepFaceLab\mainscripts\Trainer.py", line 68, in model_save
model.save()
File "D:\Program Files (x86)\Everything Deepfake\DeepFaceLab\DeepFaceLab_NVIDIA_up_to_RTX2080Ti\_internal\DeepFaceLab\models\ModelBase.py", line 393, in save
self.onSave()
File "D:\Program Files (x86)\Everything Deepfake\DeepFaceLab\DeepFaceLab_NVIDIA_up_to_RTX2080Ti\_internal\DeepFaceLab\models\Model_SAEHD\Model.py", line 759, in onSave
model.save_weights ( self.get_strpath_storage_for_file(filename) )
File "D:\Program Files (x86)\Everything Deepfake\DeepFaceLab\DeepFaceLab_NVIDIA_up_to_RTX2080Ti\_internal\DeepFaceLab\core\leras\layers\Saveable.py", line 61, in save_weights
d_dumped = pickle.dumps (d, 4)
MemoryErrorSource: iperov/DeepFaceLab