no error train stop? wandb crashed, The run stopped sending data or did not successfully start.
4*4090, lora pi0.5,这是配置文件,日志都正常,wandb 显示crashed,The run stopped sending data or did not successfully start.
TrainConfig( name="pi05_realman_eef12_lift_xiongan", model=pi0_config.Pi0Config(pi05=True, action_horizon=14, discrete_state_input=True, paligemma_variant="gemma_2b_lora", action_expert_variant="gemma_300m_lora"), data=LeRobotRealmanEEFLiftDataConfig( repo_id="dt-sean/realman_pickup_xiongan", base_config=DataConfig(prompt_from_task=True), ), batch_size=96, lr_schedule=_optimizer.CosineDecaySchedule( warmup_steps=10_000, peak_lr=5e-5, decay_steps=1_000_000, decay_lr=5e-5, ), optimizer=_optimizer.AdamW(clip_gradient_norm=1.0), ema_decay=None, weight_loader=weight_loaders.CheckpointWeightLoader("gs://openpi-assets/checkpoints/pi05_base/params"), freeze_filter=pi0_config.Pi0Config( paligemma_variant="gemma_2b_lora", action_expert_variant="gemma_300m_lora" ).get_freeze_filter(), num_train_steps=30_000, save_interval=5000, keep_period=5000, num_workers=8, ),
14:28:04.795 [I] Progress on: 9.83kit/30.0kit rate:2.0s/it remaining:11:19:50 elapsed:5:33:02 postfix:- (32166:tqdm_logging.py:145) 14:28:15.016 [I] Progress on: 9.84kit/30.0kit rate:2.1s/it remaining:11:32:39 elapsed:5:33:12 postfix:- (32166:tqdm_logging.py:145) 14:28:27.001 [I] Progress on: 9.84kit/30.0kit rate:2.0s/it remaining:11:17:41 elapsed:5:33:24 postfix:- (32166:tqdm_logging.py:145) 14:28:37.129 [I] Progress on: 9.85kit/30.0kit rate:2.0s/it remaining:11:17:27 elapsed:5:33:34 postfix:- (32166:tqdm_logging.py:145) 14:28:47.377 [I] Progress on: 9.85kit/30.0kit rate:2.1s/it remaining:11:33:40 elapsed:5:33:44 postfix:- (32166:tqdm_logging.py:145) 14:28:59.344 [I] Progress on: 9.86kit/30.0kit rate:2.0s/it remaining:11:16:54 elapsed:5:33:56 postfix:- (32166:tqdm_logging.py:145)
Source: Physical-Intelligence/openpi