[Usage] Missing "trainer_state.json" when resuming training from saved checkpoints

Describe the issue

Issue: I'm trying to reproduce the training results. As my cluster has 4-hour usage limit per use, I need to save checkpoints and resume training from them. However, resume training failed because of this error.

Command:

./scripts/v1_5/pretrain.sh

where I change the save_steps to be 500.

Log:

[Errno 2] No such file or directory: './checkpoints/llava-v1.5-7b-pretrain/checkpoint-1500/trainer_state.json'

Screenshots: I have "checkpoint-1500" folder, but it does not contain such "trainer_state.json" file.