qwen2.5-3b OOM Issue on 8x V100 GPUs

I'm running a Qwen2.5-3B grpo script on an 8-card V100 setup and keep getting OOM errors even after trying different parameter combinations. Could anyone help me out with this? Thanks a lot!

export N_GPUS=8 export BASE_MODEL=/TinyZero/Qwen2.5-3B export DATA_DIR=/TinyZero/zerodata export ROLLOUT_TP_SIZE=4 export EXPERIMENT_NAME=countdown-qwen2.5-3b export VLLM_ATTENTION_BACKEND=XFORMERS

1 python3 -m verl.trainer.main_ppo
2 algorithm.adv_estimator=grpo
3 data.train_files=$DATA_DIR/train.parquet
4 data.val_files=$DATA_DIR/test.parquet
5 data.train_batch_size=8
6 data.val_batch_size=8
7 data.max_prompt_length=256
8 data.max_response_length=512
9 actor_rollout_ref.model.path=$BASE_MODEL
10 actor_rollout_ref.actor.optim.lr=1e-6
11 actor_rollout_ref.model.use_remove_padding=True
12 actor_rollout_ref.actor.ppo_mini_batch_size=1
13 actor_rollout_ref.actor.ppo_micro_batch_size=1
14 actor_rollout_ref.actor.use_kl_loss=True
15 actor_rollout_ref.actor.kl_loss_coef=0.001
16 actor_rollout_ref.actor.kl_loss_type=low_var_kl
17 actor_rollout_ref.model.enable_gradient_checkpointing=True
18 actor_rollout_ref.actor.fsdp_config.param_offload=True
19 actor_rollout_ref.actor.fsdp_config.grad_offload=True
20 actor_rollout_ref.actor.fsdp_config.optimizer_offload=True
21 actor_rollout_ref.rollout.log_prob_micro_batch_size=2
22 actor_rollout_ref.rollout.tensor_model_parallel_size=$ROLLOUT_TP_SIZE
23 actor_rollout_ref.rollout.name=vllm
24 actor_rollout_ref.rollout.gpu_memory_utilization=0.6
25 actor_rollout_ref.rollout.n=2
26 actor_rollout_ref.ref.log_prob_micro_batch_size=2
27 actor_rollout_ref.ref.fsdp_config.param_offload=True
28 algorithm.kl_ctrl.kl_coef=0.001
29 trainer.critic_warmup=0
30 trainer.logger=['wandb']
31 +trainer.val_before_train=False
32 trainer.default_hdfs_dir=null
33 trainer.n_gpus_per_node=$N_GPUS
34 trainer.nnodes=1
35 trainer.save_freq=10
36 trainer.test_freq=10
37 trainer.project_name=TinyZero
38 trainer.experiment_name=$EXPERIMENT_NAME
39 trainer.total_epochs=15 2>&1 | tee verl_demo.log

errormessage： ray.exceptions.OutOfMemoryError: Task was killed due to the node running low on memory. Memory on the node (IP: 10.0.0.11, ID: ) where the task (task ID: , name=main_task, pid=41260, memory used=1.54GB) was running was 632.61GB / 661.38GB (0.956503), which exceeds the memory usage threshold of 0.95. Ray killed this worker (ID: ) because it was the most recently scheduled task; to see more information about memory usage on this node, use ray logs raylet.out -ip 10.0.0.11. To see the logs of the worker, use `ray logs worker-*out -ip 10.0.0.11. Top 10 memory users:

• PID MEM(GB) COMMAND
• 41679 78.93 ray::WorkerDict.actor_rollout_generate_sequences
• 41678 77.64 ray::WorkerDict.actor_rollout_generate_sequences
• 41522 77.62 ray::WorkerDict.actor_rollout_generate_sequences
• 41682 77.47 ray::WorkerDict.actor_rollout_generate_sequences
• 41680 77.25 ray::WorkerDict.actor_rollout_generate_sequences
• 41681 76.55 ray::WorkerDict.actor_rollout_generate_sequences
• 41676 75.30 ray::WorkerDict.actor_rollout_generate_sequences
• 41677 74.09 ray::WorkerDict.actor_rollout_generate_sequences
• 41260 1.54 ray::main_task
• 38690 0.46 /opt/conda/envs/zero/lib/python3.9/site-packages/ray/core/src/ray/gcs/gcs_server --log_dir=/tmp/ray/...

Refer to the documentation on how to address the out of memory issue: https://docs.ray.io/en/latest/ray-core/scheduling/ray-oom-prevention.html. Consider provisioning more memory on this node or reducing task parallelism by requesting more CPUs per task. To adjust the kill threshold, set the environment variable RAY_memory_usage_threshold when starting Ray. To disable worker killing, set the environment variable RAY_memory_monitor_refresh_ms to zero.