[HELP] Runable solution of RTX 5090 GPU + Linux Driver version + Pytorch version + Deepspeed version for LLM finetuning

Reminder

• I have read the above rules and searched the existing issues.

System Info

I want to finetune a LLM (e.g Llama-3-8B) using RTX 5090 on ubuntu 22.04.5 LTS, but failed.

I used: Linux driver: https://www.nvidia.com/en-us/drivers/details/240524/

Pytoch version: pip install --pre torch torchvision torchaudio --index-url https://download.pytorch.org/whl/nightly/cu128

Deepspeed version: 0.15.1 (or 0.16.3)

I hope someone give a possible/runable solution for Linux Driver version + Pytorch version + Deepspeed version for LLM finetuning using RTX 5090

Reproduction

Put your message here.

EXPERIMENT_NAME="Test" eval "$(/opt/miniconda3/bin/conda shell.bash hook)" cd /home/test conda activate finetune_01 export PATH=/usr/local/cuda-12.8/bin:${PATH} export LD_LIBRARY_PATH=/usr/local/cuda-12.8/lib64:${LD_LIBRARY_PATH} export LD_LIBRARY_PATH=$ROCM_PATH/lib:$LD_LIBRARY_PATH export NCCL_P2P_DISABLE=1 export NCCL_IB_DISABLE=1 export WANDB_DISABLED=true export CUDA_VISIBLE_DEVICES=0 deepspeed --num_gpus 1 --num_nodes 1 /src/train.py --stage sft --model_name_or_path "/home/z890/model/Llama-3-8B" --do_train --dataset "alpaca" --max_length 4096 --finetuning_type lora --output_dir "/home/test/output/" --overwrite_cache --per_device_train_batch_size 2 --gradient_accumulation_steps 1 --lr_scheduler_type cosine --logging_steps 1 --save_strategy steps --save_steps 5000 --learning_rate 4e-4 --num_train_epochs 17 --plot_loss --fp16 --lora_target q_proj,v_proj --lora_r 1024 --lora_alpha 2048 --lora_dropout 0.05 --preprocessing_num_workers 96 --template default --deepspeed ~/deepspeed/deepspeed_stage3.json

Others

No response