单机多卡,开启swanlab日志输出,报错:[rank0]: raise RuntimeError("No active Run. Call swanlab.init() first.")
Author: goSilverCreated Jul 30, 2026Updated Sep 6, 2026
Labelsbugpending
Reminder
- I have read the above rules and searched the existing issues.
System Info
llamafactoryversion: 0.9.6.dev0- Platform: Linux-5.15.0-25-generic-x86_64-with-glibc2.35
- Python version: 3.12.3
- PyTorch version: 2.6.0+cu126 (GPU)
- Transformers version: 5.8.0
- Datasets version: 4.0.0
- Accelerate version: 1.11.0
- PEFT version: 0.18.1
- GPU type: NVIDIA GeForce RTX 4080 SUPER
- GPU number: 3
- GPU memory: 31.48GB
- TRL version: 0.24.0
- DeepSpeed version: 0.19.3
- Git commit: 9ce6b663e9d87cd3c0cb42a1d3ff5cdfe292426d
- Default data directory: detected
Reproduction
启动命令:SWANLAB_IGNORE_RANKS=1 CUDA_VISIBLE_DEVICES=0,1,2 FORCE_TORCHRUN=1 NNODES=1 NODE_RANK=0 NPROC_PER_NODE=3 MASTER_ADDR=127.0.0.1 MASTER_PORT=29500 llamafactory-cli train /root/qwen3_4b.yaml
报错:
[INFO|integration_utils.py:2325] 2026-07-30 09:21:24,487 >> Automatic SwanLab logging enabled, to disable set os.environ["SWANLAB_MODE"] = "disabled"
[rank0]: Traceback (most recent call last):
[rank0]: File "/root/LLaMA-Factory/src/llamafactory/launcher.py", line 185, in <module>
[rank0]: run_exp()
[rank0]: File "/root/LLaMA-Factory/src/llamafactory/train/tuner.py", line 173, in run_exp
[rank0]: _training_function(config={"args": args, "callbacks": callbacks})
[rank0]: File "/root/LLaMA-Factory/src/llamafactory/train/tuner.py", line 141, in _training_function
[rank0]: run_sft(model_args, data_args, training_args, finetuning_args, generating_args, callbacks)
[rank0]: File "/root/LLaMA-Factory/src/llamafactory/train/sft/workflow.py", line 121, in run_sft
[rank0]: train_result = trainer.train(resume_from_checkpoint=training_args.resume_from_checkpoint)
[rank0]: ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[rank0]: File "/root/miniconda3/lib/python3.12/site-packages/transformers/trainer.py", line 1427, in train
[rank0]: return inner_training_loop(
[rank0]: ^^^^^^^^^^^^^^^^^^^^
[rank0]: File "/root/miniconda3/lib/python3.12/site-packages/transformers/trainer.py", line 1502, in _inner_training_loop
[rank0]: self.control = self.callback_handler.on_train_begin(args, self.state, self.control)
[rank0]: ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[rank0]: File "/root/miniconda3/lib/python3.12/site-packages/transformers/trainer_callback.py", line 492, in on_train_begin
[rank0]: return self.call_event("on_train_begin", args, state, control, **kwargs)
[rank0]: ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[rank0]: File "/root/miniconda3/lib/python3.12/site-packages/transformers/trainer_callback.py", line 545, in call_event
[rank0]: result = getattr(callback, event)(
[rank0]: ^^^^^^^^^^^^^^^^^^^^^^^^^
[rank0]: File "/root/miniconda3/lib/python3.12/site-packages/transformers/integrations/integration_utils.py", line 2390, in on_train_begin
[rank0]: self.setup(args, state, model, **kwargs)
[rank0]: File "/root/miniconda3/lib/python3.12/site-packages/transformers/integrations/integration_utils.py", line 2354, in setup
[rank0]: if self._swanlab.get_run() is None:
[rank0]: ^^^^^^^^^^^^^^^^^^^^^^^
[rank0]: File "/root/miniconda3/lib/python3.12/site-packages/swanlab/sdk/internal/run/__init__.py", line 837, in get_run
[rank0]: raise RuntimeError("No active Run. Call swanlab.init() first.")
[rank0]: RuntimeError: No active Run. Call swanlab.init() first.
AI回复:
你现在报错发生在 **rank0**,说明不是多进程抢占问题!
LLaMA Factory 内置的 SwanLabCallback 内部逻辑缺陷:
回调触发时机早于 `swanlab.init()`,调用 `swanlab.log()` 的时候 run 还未初始化,直接抛出:
`No active Run. Call swanlab.init() first.`
Others
配置文件:
### model
model_name_or_path: /root/autodl-tmp/models/Qwen/Qwen3-4B-Instruct-2507
trust_remote_code: true
### method
stage: sft
do_train: true
finetuning_type: lora
lora_rank: 8
lora_target: all
### dataset
dataset: law_train_dataset
template: qwen3
cutoff_len: 2048
max_samples: 1000000
overwrite_cache: true
preprocessing_num_workers: 16
dataloader_num_workers: 4
### output
output_dir: /root/autodl-tmp/models/Qwen/Qwen3-4B-Instruct-2507-lora-sft
logging_steps: 10
save_steps: 500
plot_loss: true
overwrite_output_dir: true
save_only_model: false
report_to: swanlab # choices: [none, wandb, tensorboard, swanlab, mlflow]
### train
per_device_train_batch_size: 8 #根据你的显存大小适当进行调整
gradient_accumulation_steps: 8
learning_rate: 1.0e-4
num_train_epochs: 10.0
lr_scheduler_type: cosine
warmup_ratio: 0.1
bf16: true
ddp_timeout: 180000000
resume_from_checkpoint: null
### eval
# eval_dataset: alpaca_en_demo
# val_size: 0.1
# per_device_eval_batch_size: 1
# eval_strategy: steps
# eval_steps: 500
Source: hiyouga/LlamaFactory