Qwen3.5-27B lora微调时128步报错oom

Reminder

• I have read the above rules and searched the existing issues.

System Info

model

model_name_or_path: /workspace/model/Qwen3.5-27B trust_remote_code: true deepspeed: examples/deepspeed/ds_z3_config.json flash_attn: auto

quantization_method: bnb quantization_type: nf4 double_quantization: true upcast_layernorm: true

freeze_vision_tower: true freeze_multi_modal_projector: true

method

stage: sft do_train: true finetuning_type: lora lora_rank: 64 lora_alpha: 64 lora_dropout: 0 lora_target: ["q_proj", "k_proj", "v_proj", "o_proj", "gate_proj", "up_proj", "down_proj", "out_proj"]

dataset

dataset: xxx #dataset: xxx template: qwen3_5 cutoff_len: 8192 preprocessing_num_workers: 16 dataloader_num_workers: 4 train_on_prompt: false enable_thinking: true

output

output_dir: saves/Qwen3.5-27B/lora/sft logging_steps: 1 save_strategy: steps save_steps: 200 plot_loss: true overwrite_output_dir: true save_only_model: false report_to: none

train

per_device_train_batch_size: 1 gradient_accumulation_steps: 4 learning_rate: 2e-4 num_train_epochs: 3.0 lr_scheduler_type: cosine weight_decay: 0.001 warmup_ratio: 0.05 bf16: true ddp_timeout: 180000000 resume_from_checkpoint: null

eval

eval_dataset: xxx

val_size: 0.01

per_device_eval_batch_size: 1

eval_strategy: steps

eval_steps: 500

训练配置如上 三张L40s 训练过程中报oom了 很奇怪 是128步 为啥会在这时候oom？

Reproduction

{'loss': '0.3436', 'grad_norm': '0.1847', 'learning_rate': '0.000168', 'epoch': '0.1276'}
{'loss': '0.2091', 'grad_norm': '0.1492', 'learning_rate': '0.0001693', 'epoch': '0.1286'}
4%|█████▌ | 128/2985 [1:32:42<33:59:01, 42.82s/it][rank0]: Traceback (most recent call last): [rank0]: File "/workspace/26_04_07_llamafactory/LlamaFactory/src/llamafactory/launcher.py", line 185, in [rank0]: run_exp() [rank0]: File "/workspace/26_04_07_llamafactory/LlamaFactory/src/llamafactory/train/tuner.py", line 139, in run_exp [rank0]: _training_function(config={"args": args, "callbacks": callbacks}) [rank0]: File "/workspace/26_04_07_llamafactory/LlamaFactory/src/llamafactory/train/tuner.py", line 107, in _training_function [rank0]: run_sft(model_args, data_args, training_args, finetuning_args, generating_args, callbacks) [rank0]: File "/workspace/26_04_07_llamafactory/LlamaFactory/src/llamafactory/train/sft/workflow.py", line 140, in run_sft [rank0]: train_result = trainer.train(resume_from_checkpoint=training_args.resume_from_checkpoint) [rank0]: ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ [rank0]: File "/workspace/26_04_07_llamafactory/myenv/lib/python3.11/site-packages/transformers/trainer.py", line 1412, in train [rank0]: return inner_training_loop( [rank0]: ^^^^^^^^^^^^^^^^^^^^ [rank0]: File "/workspace/26_04_07_llamafactory/myenv/lib/python3.11/site-packages/transformers/trainer.py", line 1742, in _inner_training_loop [rank0]: tr_loss_step = self.training_step(model, inputs, num_items_in_batch) [rank0]: ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ [rank0]: File "/workspace/26_04_07_llamafactory/myenv/lib/python3.11/site-packages/transformers/trainer.py", line 1951, in training_step [rank0]: loss = self.compute_loss(model, inputs, num_items_in_batch=num_items_in_batch) [rank0]: ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ [rank0]: File "/workspace/26_04_07_llamafactory/LlamaFactory/src/llamafactory/train/sft/trainer.py", line 162, in compute_loss [rank0]: return super().compute_loss(model, inputs, *args, **kwargs) [rank0]: ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ [rank0]: File "/workspace/26_04_07_llamafactory/myenv/lib/python3.11/site-packages/transformers/trainer.py", line 2022, in compute_loss [rank0]: outputs = model(**inputs) [rank0]: ^^^^^^^^^^^^^^^ [rank0]: File "/workspace/26_04_07_llamafactory/myenv/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1779, in _wrapped_call_impl [rank0]: return self._call_impl(*args, **kwargs) [rank0]: ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ [rank0]: File "/workspace/26_04_07_llamafactory/myenv/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1790, in _call_impl [rank0]: return forward_call(*args, **kwargs) [rank0]: ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ [rank0]: File "/workspace/26_04_07_llamafactory/myenv/lib/python3.11/site-packages/deepspeed/utils/nvtx.py", line 20, in wrapped_fn [rank0]: ret_val = func(*args, **kwargs) [rank0]: ^^^^^^^^^^^^^^^^^^^^^ [rank0]: File "/workspace/26_04_07_llamafactory/myenv/lib/python3.11/site-packages/deepspeed/runtime/engine.py", line 2358, in forward [rank0]: loss = self.module(*inputs, **kwargs) [rank0]: ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ [rank0]: File "/workspace/26_04_07_llamafactory/myenv/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1779, in _wrapped_call_impl [rank0]: return self._call_impl(*args, **kwargs) [rank0]: ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ [rank0]: File "/workspace/26_04_07_llamafactory/myenv/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1885, in _call_impl [rank0]: return inner() [rank0]: ^^^^^^^ [rank0]: File "/workspace/26_04_07_llamafactory/myenv/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1833, in inner [rank0]: result = forward_call(*args, **kwargs) [rank0]: ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ [rank0]: File "/workspace/26_04_07_llamafactory/myenv/lib/python3.11/site-packages/peft/peft_model.py", line 1923, in forward [rank0]: return self.base_model( [rank0]: ^^^^^^^^^^^^^^^^ [rank0]: File "/workspace/26_04_07_llamafactory/myenv/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1779, in _wrapped_call_impl [rank0]: return self._call_impl(*args, **kwargs) [rank0]: ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ [rank0]: File "/workspace/26_04_07_llamafactory/myenv/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1885, in _call_impl [rank0]: return inner() [rank0]: ^^^^^^^ [rank0]: File "/workspace/26_04_07_llamafactory/myenv/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1833, in inner [rank0]: result = forward_call(*args, **kwargs) [rank0]: ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ [rank0]: File "/workspace/26_04_07_llamafactory/myenv/lib/python3.11/site-packages/peft/tuners/tuners_utils.py", line 311, in forward [rank0]: return self.model.forward(*args, **kwargs) [rank0]: ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ [rank0]: File "/workspace/26_04_07_llamafactory/myenv/lib/python3.11/site-packages/transformers/utils/generic.py", line 841, in wrapper [rank0]: output = func(self, *args, **kwargs) [rank0]: ^^^^^^^^^^^^^^^^^^^^^^^^^^^ [rank0]: File "/workspace/26_04_07_llamafactory/myenv/lib/python3.11/site-packages/transformers/models/qwen3_5/modeling_qwen3_5.py", line 1960, in forward [rank0]: loss = self.loss_function(logits=logits, labels=labels, vocab_size=self.config.text_config.vocab_size) [rank0]: ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ [rank0]: File "/workspace/26_04_07_llamafactory/myenv/lib/python3.11/site-packages/transformers/loss/loss_utils.py", line 66, in ForCausalLMLoss [rank0]: loss = fixed_cross_entropy(logits, shift_labels, num_items_in_batch, ignore_index, **kwargs) [rank0]: ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ [rank0]: File "/workspace/26_04_07_llamafactory/myenv/lib/python3.11/site-packages/transformers/loss/loss_utils.py", line 36, in fixed_cross_entropy [rank0]: loss = nn.functional.cross_entropy(source, target, ignore_index=ignore_index, reduction=reduction) [rank0]: ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ [rank0]: File "/workspace/26_04_07_llamafactory/myenv/lib/python3.11/site-packages/torch/nn/functional.py", line 3504, in cross_entropy [rank0]: return torch._C._nn.cross_entropy_loss( [rank0]: ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ [rank0]: torch.OutOfMemoryError: CUDA out of memory. Tried to allocate 7.08 GiB. GPU 0 has a total capacity of 47.38 GiB of which 6.24 GiB is free. Process 1253424 has 41.13 GiB memory in use. Of the allocated memory 38.79 GiB is allocated by PyTorch, and 1.66 GiB is reserved by PyTorch but unallocated. If reserved but unallocated memory is large try setting PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True to avoid fragmentation. See documentation for Memory Management (https://docs.pytorch.org/docs/stable/notes/cuda.html#optimizing-memory-usage-with-pytorch-cuda-alloc-conf) 4%|█████▌ | 128/2985 [1:33:02<34:36:54, 43.62s/it]

Others

No response