使用 sample_packing_eff_est 时, total_num_steps 计算结果不正确
作者: sageof6path创建于 2025年9月11日更新于 2026年7月17日
标签bugwaiting for reporter
Please check that this issue hasn't been reported before. - [x] I searched previous Bug Reports didn't find any similar reports. ### Expected Behavior In https://GitHub.com/axolotl-ai-cloud/axolotl/blob/main/src/axolotl/utils/trainer.py function calculate_total_num_steps should always return correct value for training step. ### Current behaviour In https://GitHub.com/axolotl-ai-cloud/axolotl/blob/main/src/axolotl/utils/trainer.py When sample_packing_eff_est value is set, we use following formula to calculate steps: total_num_steps = ( # match count to len est in dataloader int(math.floor( 0.99 * cfg.total_num_tokens / cfg.sample_packing_eff_est / cfg.sequence_len // cfg.batch_size ) - 1 ) * cfg.num_epochs * cfg.context_parallel_size * cfg.tensor_parallel_size ) By Running following commands: Docker run --gpus '"all"' --shm-size 64gb --rm -it --entrypoint sh axolotlai/axolotl:main-20250911-py3.11-cu126-2.6.0 sed -i 's#LOG.debug#LOG.info#' /workspace/axolotl/src/axolotl/utils/trainer.py axolotl train examples/LLaMA-3/lora-1b.yml We see following log Saving the dataset (208/208 shards): 100%|█████████████████████| 54568/54568 [00:02<00:00, 22406.82 examples/s] [2025-09-11 10:38:27,267] [INFO] [axolotl.utils.samplers.multipack.calc_min_len:434] [PID:428] [RANK:0] gather_len_batches: [230, 232, 231, 234, 230, 230, 227, 234] [2025-09-11 10:38:27,268] [INFO] [axolotl.utils.trainer.calculate_total_num_steps:481] [PID:428] [RANK:0] data_loader_len: 14 [2025-09-11 10:38:27,279] [INFO] [axolotl.utils.trainer.calc_sample_packing_eff_est:497] [PID:428] [RANK:0] sample_packing_eff_est across ranks: [0.9955014586448669, 0.993775486946106, 0.9941712617874146, 0.9956336617469788, 0.9948931932449341, 0.9948210120201111, 0.9928134679794312, 0.9942992329597473] [2025-09-11 10:38:27,279] [INFO] [axolotl.utils.trainer.calculate_total_num_steps:509] [PID:428] [RANK:0] sample_packing_eff_est: None [2025-09-11 10:38:27,279] [INFO] [axolotl.utils.trainer.calculate_total_num_steps:520] [PID:428] [RANK:0] total_num_steps: 14 [2025-09-11 10:38:27,337] [INFO] [axolotl.utils.trainer.calculate_total_num_steps:404] [PID:428] [RANK:0] total_num_tokens: 8_487_290 [2025-09-11 10:38:27,519] [INFO] [axolotl.utils.trainer.calculate_total_num_steps:422] [PID:428] [RANK:0] total_supervised_tokens: 5_561_191 [2025-09-11 10:38:31,274] [INFO] [axolotl.utils.samplers.multipack.calc_min_len:434] [PID:428] [RANK:0] gather_len_batches: [2079, 2077, 2078, 2075, 2079, 2079, 2082, 2075] [2025-09-11 10:38:31,274] [INFO] [axolotl.utils.trainer.calculate_total_num_steps:481] [PID:428] [RANK:0] data_loader_len: 129 …
内容来源: axolotl-ai-cloud/axolotl