Baike.dev
Log in
Back to tool/Back to issues
#3152·axolotl

total_num_steps calculation is incorrect with sample_packing_eff_est

Author: sageof6pathCreated Sep 11, 2025Updated Jul 17, 2026
Labelsbugwaiting for reporter

Please check that this issue hasn't been reported before.

  • I searched previous Bug Reports didn't find any similar reports.

Expected Behavior

In https://github.com/axolotl-ai-cloud/axolotl/blob/main/src/axolotl/utils/trainer.py function calculate_total_num_steps should always return correct value for training step.

Current behaviour

In https://github.com/axolotl-ai-cloud/axolotl/blob/main/src/axolotl/utils/trainer.py When sample_packing_eff_est value is set, we use following formula to calculate steps:

total_num_steps = (
                # match count to len est in dataloader
                int(
                    math.floor(
                        0.99
                        * cfg.total_num_tokens
                        / cfg.sample_packing_eff_est
                        / cfg.sequence_len
                        // cfg.batch_size
                    )
                    - 1
                )
                * cfg.num_epochs
                * cfg.context_parallel_size
                * cfg.tensor_parallel_size
            )

By Running following commands:

bash
docker run --gpus '"all"' --shm-size 64gb --rm -it --entrypoint sh axolotlai/axolotl:main-20250911-py3.11-cu126-2.6.0

sed -i 's#LOG.debug#LOG.info#' /workspace/axolotl/src/axolotl/utils/trainer.py

axolotl train examples/llama-3/lora-1b.yml

We see following log

Saving the dataset (208/208 shards): 100%|█████████████████████| 54568/54568 [00:02<00:00, 22406.82 examples/s]
[2025-09-11 10:38:27,267] [INFO] [axolotl.utils.samplers.multipack.calc_min_len:434] [PID:428] [RANK:0] gather_len_batches: [230, 232, 231, 234, 230, 230, 227, 234]
[2025-09-11 10:38:27,268] [INFO] [axolotl.utils.trainer.calculate_total_num_steps:481] [PID:428] [RANK:0] data_loader_len: 14
[2025-09-11 10:38:27,279] [INFO] [axolotl.utils.trainer.calc_sample_packing_eff_est:497] [PID:428] [RANK:0] sample_packing_eff_est across ranks: [0.9955014586448669, 0.993775486946106, 0.9941712617874146, 0.9956336617469788, 0.9948931932449341, 0.9948210120201111, 0.9928134679794312, 0.9942992329597473]
[2025-09-11 10:38:27,279] [INFO] [axolotl.utils.trainer.calculate_total_num_steps:509] [PID:428] [RANK:0] sample_packing_eff_est: None
[2025-09-11 10:38:27,279] [INFO] [axolotl.utils.trainer.calculate_total_num_steps:520] [PID:428] [RANK:0] total_num_steps: 14
[2025-09-11 10:38:27,337] [INFO] [axolotl.utils.trainer.calculate_total_num_steps:404] [PID:428] [RANK:0] total_num_tokens: 8_487_290
[2025-09-11 10:38:27,519] [INFO] [axolotl.utils.trainer.calculate_total_num_steps:422] [PID:428] [RANK:0] `total_supervised_tokens: 5_561_191`
[2025-09-11 10:38:31,274] [INFO] [axolotl.utils.samplers.multipack.calc_min_len:434] [PID:428] [RANK:0] gather_len_batches: [2079, 2077, 2078, 2075, 2079, 2079, 2082, 2075]
[2025-09-11 10:38:31,274] [INFO] [axolotl.utils.trainer.calculate_total_num_steps:481] [PID:428] [RANK:0] data_loader_len: 129
[2025-09-11 10:38:31,277] [INFO] [axolotl.utils.trainer.calc_sample_packing_eff_est:497] [PID:428] [RANK:0] sample_packing_eff_est across ranks: [0.9966773986816406, 0.996871292591095, 0.9968258142471313, 0.9966647624969482, 0.996744692325592, 0.9967526793479919, 0.9964901208877563, 0.9968152046203613]
[2025-09-11 10:38:31,277] [INFO] [axolotl.utils.trainer.calculate_total_num_steps:509] [PID:428] [RANK:0] sample_packing_eff_est: 1.0
[2025-09-11 10:38:31,277] [INFO] [axolotl.utils.trainer.calculate_total_num_steps:520] [PID:428] [RANK:0] total_num_steps: 129
[2025-09-11 10:38:31,280] [INFO] [axolotl.utils.data.sft._prepare_standard_dataset:121] [PID:428] [RANK:0] Maximum number of steps set at 129
[2025-09-11 10:38:31,874] [INFO] [axolotl.loaders.tokenizer.load_tokenizer:300] [PID:428] [RANK:0] No Chat template selected. Consider adding a chat template for easier inference.
`torch_dtype` is deprecated! Use `dtype` instead!
`torch_dtype` is deprecated! Use `dtype` instead!
`torch_dtype` is deprecated! Use `dtype` instead!
`torch_dtype` is deprecated! Use `dtype` instead!
`torch_dtype` is deprecated! Use `dtype` instead!
`torch_dtype` is deprecated! Use `dtype` instead!
`torch_dtype` is deprecated! Use `dtype` instead!
[2025-09-11 10:38:31,956] [INFO] [axolotl.monkeypatch.transformers.trainer_loss_calc.patch_evaluation_loop:87] [PID:428] [RANK:0] Patched Trainer.evaluation_loop with nanmean loss calculation
[2025-09-11 10:38:31,957] [INFO] [axolotl.monkeypatch.transformers.trainer_loss_calc.patch_maybe_log_save_evaluate:138] [PID:428] [RANK:0] Patched Trainer._maybe_log_save_evaluate with nanmean loss calculation
[2025-09-11 10:38:31,958] [INFO] [axolotl.loaders.patch_manager._apply_multipack_patches:279] [PID:428] [RANK:0] Applying multipack dataloader patch for sample packing...
`torch_dtype` is deprecated! Use `dtype` instead!
model.safetensors: 100%|███████████████████████████████████████████████████| 2.47G/2.47G [00:03<00:00, 777MB/s]
generation_config.json: 100%|█████████████████████████████████████████████████| 186/186 [00:00<00:00, 1.63MB/s]
[2025-09-11 10:38:36,457] [INFO] [axolotl.loaders.model._configure_embedding_dtypes:345] [PID:428] [RANK:0] Converting modules to torch.bfloat16
trainable params: 11,272,192 || all params: 1,247,086,592 || trainable%: 0.9039
df: /root/.triton/autotune: No such file or directory
[2025-09-11 10:38:39,277] [INFO] [axolotl.train.save_initial_configs:414] [PID:428] [RANK:0] Pre-saving adapter config to ./outputs/lora-out...
[2025-09-11 10:38:39,277] [INFO] [axolotl.train.save_initial_configs:418] [PID:428] [RANK:0] Pre-saving tokenizer to ./outputs/lora-out...
[2025-09-11 10:38:39,430] [INFO] [axolotl.train.save_initial_configs:423] [PID:428] [RANK:0] Pre-saving model config to ./outputs/lora-out...
[2025-09-11 10:38:39,432] [INFO] [axolotl.train.execute_training:203] [PID:428] [RANK:0] Starting trainer...
[2025-09-11 10:38:45,452] [INFO] [axolotl.utils.samplers.multipack.calc_min_len:434] [PID:428] [RANK:0] gather_len_batches: [2079, 2077, 2078, 2075, 2079, 2079, 2082, 2075]
  0%|                                                                                  | 0/129 [00:00<?, ?it/s][2025-09-11 10:38:49,857] [INFO] [axolotl.utils.samplers.multipack.calc_min_len:434] [PID:428] [RANK:0] gather_len_batches: [230, 232, 231, 234, 230, 230, 227, 234]
{'eval_loss': 1.3563038110733032, 'eval_runtime': 7.1942, 'eval_samples_per_second': 758.526, 'eval_steps_per_second': 47.538, 'memory/max_active (GiB)': 7.47, 'memory/max_allocated (GiB)': 7.47, 'memory/device_reserved (GiB)': 7.51, 'epoch': 0}
{'loss': 1.3838, 'grad_norm': 0.4009654223918915, 'learning_rate': 0.0, 'memory/max_active (GiB)': 8.78, 'memory/max_allocated (GiB)': 8.78, 'memory/device_reserved (GiB)': 10.85, 'tokens_per_second_per_gpu': 83455.75, 'epoch': 0.01}

sample_packing_eff_est = 1.0 total_num_steps = 129 total_num_tokens = 8487290 sequence_len = 2048 num_epochs = 1 batch_size = num of gpus * gradient_accumulation_stepsmicro_batch_size = 82*2 = 32 context_parallel_size = 1 tensor_parallel_size = 1

total steps by above formula = int(math.floor(0.99 * 8487290 / 1.0 / 2048 / 32) - 1) * 1 * 1 * 1 = int(math.floor(128.210710144)-1) = 127 = 127

this is lower than 129

Steps to reproduce

Run following commands

bash
docker run --gpus '"all"' --shm-size 64gb --rm -it --entrypoint sh axolotlai/axolotl:main-20250911-py3.11-cu126-2.6.0

sed -i 's#LOG.debug#LOG.info#' /workspace/axolotl/src/axolotl/utils/trainer.py

axolotl train examples/llama-3/lora-1b.yml

Config yaml

yaml
https://github.com/axolotl-ai-cloud/axolotl/blob/1b53c49e1a8408ff209ae72a480681d18f7f8c81/examples/llama-3/lora-1b.yml

Possible solution

No response

Which Operating Systems are you using?

  • Linux
  • macOS
  • Windows

Python Version

3.10

axolotl branch-commit

main/9406c0c

Acknowledgements

  • My issue title is concise, descriptive, and in title casing.
  • I have searched the existing issues to make sure this bug has not been reported yet.
  • I am using the latest version of axolotl.
  • I have provided enough information for the maintainers to reproduce and diagnose the issue.

Source: axolotl-ai-cloud/axolotl

View original on GitHubView discussion on GitHub