CUDA error: out of memory in WSL with 24G VRAM while 2/3 was still left unused

Author: ja3592Created Feb 22, 2025Updated Sep 28, 2026

I have fine-tuned the DeepSeek-R1-Distill-LLama-8B model using a medical dataset, following the methods found online. They used a T4 GPU with 16GB VRAM, and I attempted to replicate this locally. Here are the details of my work:

I obtained the pip command using a tool and installed Unsloth in WSL-Ubuntu（Torch: 2.3.1+cu121. CUDA: 8.6. CUDA Toolkit: 12.1. Triton: 2.3.1）.

wget -qO- https://raw.githubusercontent.com/unslothai/unsloth/main/unsloth/_auto_install.py | python - pip install "unsloth[cu121-ampere-torch230] @ git+https://github.com/unslothai/unsloth.git"

I downloaded the model and dataset, and then ran the fine-tuning code. Everything went smoothly until the training phase, where I encountered an out-of-memory error. Others online have successfully run this with a 16GB GPU, but I have a 24GB GPU in WSL, yet I still faced memory issues. The resource manager showed that the dedicated GPU memory usage was 8.7GB/24GB, while the shared GPU memory usage was 1.3GB/15.9GB (which is abnormal, as the shared memory should not be used in advance).

When I reduced the max_seq_length from 2048 to 1024, the training ran without errors, but the loss decreased very slowly, eventually stabilizing around 7.

Below is the code and output for fine-tuning the model using Unsloth. I would appreciate it if experts could help analyze the issue and provide a solution.

from unsloth import FastLanguageModel import torch max_seq_length = 2048 dtype = None load_in_4bit = True

Unsloth: Will patch your computer to enable 2x faster free finetuning. Unsloth Zoo will now patch everything to make training faster! [2025-02-21 11:01:46,935] [INFO] [real_accelerator.py:219:get_accelerator] Setting ds_accelerator to cuda (auto detect)

import wandb

wandb.login(key="*******") run = wandb.init( project='my fint-tune on deepseek r1 with medical data', job_type="training", anonymous="allow" )

model, tokenizer = FastLanguageModel.from_pretrained( model_name = "/home/finetune/unsloth/ds_llama_8/DeepSeek-R1-Distill-Llama-8B", max_seq_length = max_seq_length, dtype = dtype, load_in_4bit = load_in_4bit, )

==((====))== Unsloth 2025.2.15: Fast Llama patching. Transformers: 4.49.0. \ /| GPU: NVIDIA GeForce RTX 3090 Ti. Max memory: 23.988 GB. Platform: Linux. O^O/ _/ \ Torch: 2.3.1+cu121. CUDA: 8.6. CUDA Toolkit: 12.1. Triton: 2.3.1 \ / Bfloat16 = TRUE. FA [Xformers = 0.0.27. FA2 = True] "-____-" Free Apache license: http://github.com/unslothai/unsloth Unsloth: Fast downloading is enabled - ignore downloading bars which are red colored! /home/finetune/unsloth/ds_llama_8/DeepSeek-R1-Distill-Llama-8B does not have a padding token! Will use pad_token = <|finetune_right_pad_id|>.

model = FastLanguageModel.get_peft_model( model, r=16, target_modules=[ "q_proj", "k_proj", "v_proj", "o_proj", "gate_proj", "up_proj", "down_proj", ], lora_alpha=16, lora_dropout=0, bias="none", use_gradient_checkpointing="unsloth", # True or "unsloth" for very long context random_state=3407, use_rslora=False, loftq_config=None, )

Unsloth 2025.2.15 patched 32 layers with 32 QKV layers, 32 O layers and 32 MLP layers.

train_prompt_style = """Below is an instruction that describes a task, paired with an input that provides further context. Write a response that appropriately completes the request. Before answering, think carefully about the question and create a step-by-step chain of thoughts to ensure a logical and accurate response.

Instruction:

You are a medical expert with advanced knowledge in clinical reasoning, diagnostics, and treatment planning. Please answer the following medical question.

Question:

{}

Response:

{} {}"""

EOS_TOKEN = tokenizer.eos_token # Must add EOS_TOKEN

def formatting_prompts_func(examples): inputs = examples["Question"] cots = examples["Complex_CoT"] outputs = examples["Response"] texts = [] for input, cot, output in zip(inputs, cots, outputs): text = train_prompt_style.format(input, cot, output) + EOS_TOKEN texts.append(text) return { "text": texts, }

from datasets import load_dataset

dataset = load_dataset("/home/finetune/unsloth/ds_llama_8/data", "en",split = "train[0:500]") dataset = dataset.map(formatting_prompts_func, batched = True,) dataset["text"][0]

from trl import SFTTrainer from transformers import TrainingArguments from unsloth import is_bfloat16_supported import os #os.environ['CUDA_LAUNCH_BLOCKING'] = '1' trainer = SFTTrainer( model=model, tokenizer=tokenizer, train_dataset=dataset, dataset_text_field="text", max_seq_length=max_seq_length, dataset_num_proc=2, args=TrainingArguments( per_device_train_batch_size=2, gradient_accumulation_steps=4, # Use num_train_epochs = 1, warmup_ratio for full training runs! warmup_steps=5, max_steps=60, learning_rate=2e-4, fp16=not is_bfloat16_supported(), bf16=is_bfloat16_supported(), logging_steps=10, optim="adamw_8bit", weight_decay=0.01, lr_scheduler_type="linear", seed=3407, output_dir="outputs", #gradient_checkpointing=True, # 启用梯度检查点 ), )

trainer_stats = trainer.train()

==((====))== Unsloth - 2x faster free finetuning | Num GPUs = 1 \ /| Num examples = 500 | Num Epochs = 1 O^O/ _/ \ Batch size per device = 2 | Gradient Accumulation steps = 4 \ / Total batch size = 8 | Total steps = 60 "-____-" Number of trainable parameters = 41,943,040 [34m[1mwandb[0m: [33mWARNING[0m The run_name is currently set to the same value as TrainingArguments.output_dir. If this was not intended, please specify a different run name by setting the TrainingArguments.run_name parameter.

{ "name": "RuntimeError", "message": "CUDA error: out of memory CUDA kernel errors might be asynchronously reported at some other API call, so the stacktrace below might be incorrect. For debugging consider passing CUDA_LAUNCH_BLOCKING=1. Compile with TORCH_USE_CUDA_DSA to enable device-side assertions. ", "stack": "--------------------------------------------------------------------------- RuntimeError Traceback (most recent call last) Cell In[11], line 2 1 #torch.cuda.empty_cache() ----> 2 trainer_stats = trainer.train()

File ~/anaconda3/envs/chatglm-4-2.3/lib/python3.11/site-packages/transformers/trainer.py:2241, in Trainer.train(self, resume_from_checkpoint, trial, ignore_keys_for_eval, **kwargs) 2239 hf_hub_utils.enable_progress_bars() 2240 else: -> 2241 return inner_training_loop( 2242 args=args, 2243 resume_from_checkpoint=resume_from_checkpoint, 2244 trial=trial, 2245 ignore_keys_for_eval=ignore_keys_for_eval, 2246 )

File :329, in _fast_inner_training_loop(self, batch_size, args, resume_from_checkpoint, trial, ignore_keys_for_eval)

File :31, in _unsloth_training_step(self, model, inputs, num_items_in_batch)

File ~/finetune/unsloth/ds_llama_8/unsloth_compiled_cache/UnslothSFTTrainer.py:716, in _UnslothSFTTrainer.compute_loss(self, model, inputs, return_outputs, num_items_in_batch) 715 def compute_loss(self, model, inputs, return_outputs = False, num_items_in_batch = None): --> 716 outputs = super().compute_loss( 717 model, 718 inputs, 719 return_outputs = return_outputs, 720 num_items_in_batch = num_items_in_batch, 721 ) 722 return outputs

File ~/anaconda3/envs/chatglm-4-2.3/lib/python3.11/site-packages/unsloth/models/_utils.py:1077, in _unsloth_pre_compute_loss(self, model, inputs, *args, **kwargs) 1071 logger.warning_once( 1072 f"Unsloth: Not an error, but {name} does not accept num_items_in_batch.
"\ 1073 "Using gradient accumulation will be very slightly less accurate.
"\ 1074 "Read more on gradient accumulation issues here: https://unsloth.ai/blog/gradient\" 1075 ) 1076 pass -> 1077 return self._old_compute_loss(model, inputs, *args, **kwargs)

File ~/anaconda3/envs/chatglm-4-2.3/lib/python3.11/site-packages/transformers/trainer.py:3759, in Trainer.compute_loss(self, model, inputs, return_outputs, num_items_in_batch) 3757 loss_kwargs["num_items_in_batch"] = num_items_in_batch 3758 inputs = {**inputs, **loss_kwargs} -> 3759 outputs = model(**inputs) 3760 # Save past state if it exists 3761 # TODO: this needs to be fixed and made cleaner later. 3762 if self.args.past_index >= 0:

File ~/anaconda3/envs/chatglm-4-2.3/lib/python3.11/site-packages/torch/nn/modules/module.py:1532, in Module._wrapped_call_impl(self, *args, **kwargs) 1530 return self._compiled_call_impl(*args, **kwargs) # type: ignore[misc] 1531 else: -> 1532 return self._call_impl(*args, **kwargs)

File ~/anaconda3/envs/chatglm-4-2.3/lib/python3.11/site-packages/torch/nn/modules/module.py:1541, in Module._call_impl(self, *args, **kwargs) 1536 # If we don't have any hooks, we want to skip the rest of the logic in 1537 # this function, and just call forward. 1538 if not (self._backward_hooks or self._backward_pre_hooks or self._forward_hooks or self._forward_pre_hooks 1539 or _global_backward_pre_hooks or _global_backward_hooks 1540 or _global_forward_hooks or _global_forward_pre_hooks): -> 1541 return forward_call(*args, **kwargs) 1543 try: 1544 result = None

File ~/anaconda3/envs/chatglm-4-2.3/lib/python3.11/site-packages/accelerate/utils/operations.py:819, in convert_outputs_to_fp32..forward(*args, **kwargs) 818 def forward(*args, **kwargs): --> 819 return model_forward(*args, **kwargs)

File ~/anaconda3/envs/chatglm-4-2.3/lib/python3.11/site-packages/accelerate/utils/operations.py:807, in ConvertOutputsToFp32.call(self, *args, **kwargs) 806 def call(self, *args, **kwargs): --> 807 return convert_to_fp32(self.model_forward(*args, **kwargs))

File ~/anaconda3/envs/chatglm-4-2.3/lib/python3.11/site-packages/torch/amp/autocast_mode.py:16, in autocast_decorator..decorate_autocast(*args, **kwargs) 13 @functools.wraps(func) 14 def decorate_autocast(*args, **kwargs): 15 with autocast_instance: ---> 16 return func(*args, **kwargs)

File ~/anaconda3/envs/chatglm-4-2.3/lib/python3.11/site-packages/torch/_compile.py:24, in _disable_dynamo..inner(*args, **kwargs) 20 @functools.wraps(fn) 21 def inner(*args, **kwargs): 22 import torch._dynamo ---> 24 return torch._dynamo.disable(fn, recursive)(*args, **kwargs)

File ~/anaconda3/envs/chatglm-4-2.3/lib/python3.11/site-packages/torch/_dynamo/eval_frame.py:451, in _TorchDynamoContext.call.._fn(*args, **kwargs) 449 prior = set_eval_frame(callback) 450 try: --> 451 return fn(*args, **kwargs) 452 finally: 453 set_eval_frame(prior)

File ~/anaconda3/envs/chatglm-4-2.3/lib/python3.11/site-packages/unsloth/models/llama.py:1216, in PeftModelForCausalLM_fast_forward(self, input_ids, causal_mask, attention_mask, inputs_embeds, labels, output_attentions, output_hidden_states, return_dict, task_ids, num_logits_to_keep, logits_to_keep, **kwargs) 1200 @torch._disable_dynamo 1201 def PeftModelForCausalLM_fast_forward( 1202 self, (...) 1214 **kwargs, 1215 ): -> 1216 return self.base_model( 1217 input_ids = input_ids, 1218 causal_mask = causal_mask, 1219 attention_mask = attention_mask, 1220 inputs_embeds = inputs_embeds, 1221 labels = labels, 1222 output_attentions = output_attentions, 1223 output_hidden_states = output_hidden_states, 1224 return_dict = return_dict, 1225 num_logits_to_keep = num_logits_to_keep, 1226 logits_to_keep = logits_to_keep, 1227 **kwargs, 1228 )

File ~/anaconda3/envs/chatglm-4-2.3/lib/python3.11/site-packages/peft/tuners/tuners_utils.py:197, in BaseTuner.forward(self, *args, **kwargs) 196 def forward(self, *args: Any, **kwargs: Any): --> 197 return self.model.forward(*args, **kwargs)

File ~/anaconda3/envs/chatglm-4-2.3/lib/python3.11/site-packages/unsloth/models/llama.py:1061, in CausalLM_fast_forward.._CausalLM_fast_forward(self, input_ids, causal_mask, attention_mask, position_ids, past_key_values, inputs_embeds, labels, use_cache, output_attentions, output_hidden_states, return_dict, num_logits_to_keep, logits_to_keep, *args, **kwargs) 1059 # decoder outputs consists of (dec_features, layer_state, dec_hidden, dec_attn) 1060 self.model._has_no_labels = labels is None -> 1061 outputs = self.model( 1062 input_ids = input_ids, 1063 causal_mask = causal_mask, 1064 attention_mask = attention_mask, 1065 position_ids = position_ids, 1066 past_key_values = past_key_values, 1067 inputs_embeds = inputs_embeds, 1068 use_cache = use_cache, 1069 output_attentions = output_attentions, 1070 output_hidden_states = output_hidden_states, 1071 return_dict = return_dict, 1072 ) 1073 pass 1074 hidden_states = outputs[0]

File ~/anaconda3/envs/chatglm-4-2.3/lib/python3.11/site-packages/unsloth/models/llama.py:853, in LlamaModel_fast_forward(self, input_ids, causal_mask, attention_mask, position_ids, past_key_values, inputs_embeds, use_cache, output_attentions, output_hidden_states, return_dict, *args, **kwargs) 850 pass 852 if offloaded_gradient_checkpointing: --> 853 hidden_states = Unsloth_Offloaded_Gradient_Checkpointer.apply( 854 decoder_layer, 855 hidden_states, 856 mask, 857 attention_mask, 858 position_ids, 859 past_key_values, 860 output_attentions, 861 use_cache, 862 None, 863 position_embeddings, 864 )[0] 866 elif gradient_checkpointing: 867 def create_custom_forward(module):

File ~/anaconda3/envs/chatglm-4-2.3/lib/python3.11/site-packages/torch/autograd/function.py:598, in Function.apply(cls, *args, **kwargs) 595 if not torch._C._are_functorch_transforms_active(): 596 # See NOTE: [functorch vjp and autograd interaction] 597 args = _functorch.utils.unwrap_dead_wrappers(args) --> 598 return super().apply(*args, **kwargs) # type: ignore[misc] 600 if not is_setup_ctx_defined: 601 raise RuntimeError( 602 "In order to use an autograd.Function with functorch transforms " 603 "(vmap, grad, jvp, jacrev, ...), it must override the setup_context " 604 "staticmethod. For more details, please see " 605 "https://pytorch.org/docs/master/notes/extending.func.html\" 606 )

File ~/anaconda3/envs/chatglm-4-2.3/lib/python3.11/site-packages/torch/cuda/amp/autocast_mode.py:115, in custom_fwd..decorate_fwd(*args, **kwargs) 113 if cast_inputs is None: 114 args[0]._fwd_used_autocast = torch.is_autocast_enabled() --> 115 return fwd(*args, **kwargs) 116 else: 117 autocast_context = torch.is_autocast_enabled()

File ~/anaconda3/envs/chatglm-4-2.3/lib/python3.11/site-packages/unsloth_zoo/gradient_checkpointing.py:145, in Unsloth_Offloaded_Gradient_Checkpointer.forward(ctx, forward_function, hidden_states, *args) 142 @staticmethod 143 @torch_amp_custom_fwd 144 def forward(ctx, forward_function, hidden_states, *args): --> 145 saved_hidden_states = hidden_states.to("cpu", non_blocking = True) 146 with torch.no_grad(): 147 output = forward_function(hidden_states, *args)

RuntimeError: CUDA error: out of memory CUDA kernel errors might be asynchronously reported at some other API call, so the stacktrace below might be incorrect. For debugging consider passing CUDA_LAUNCH_BLOCKING=1. Compile with TORCH_USE_CUDA_DSA to enable device-side assertions. " }

Thank you!

Source: unslothai/unsloth

View original on GitHub View discussion on GitHub