FileExistsError on Windows at end of training when optimization.steps is a multiple of checkpoints.interval

Summary

On Windows, a training run that completes successfully crashes at the very end with FileExistsError whenever optimization.steps is an exact multiple of checkpoints.interval.

In that case train() saves a checkpoint twice at the final step:

• the periodic save inside the loop (trainer.py#L242-L248, fires because global_step % interval == 0), and
• the unconditional save after the loop (trainer.py#L323).

Both run at the same global_step, so _save_training_state() writes training_state_step_XXXXX.pt twice. The move-into-place uses tmp_path.rename(state_path) (trainer.py#L1039). pathlib.Path.rename calls os.rename, which on Windows refuses to overwrite an existing target (WinError 183) while on POSIX it replaces silently — so the second save is harmless on Linux and fatal on Windows.

No data is lost: the weights and training state are saved correctly by the first call before the crash. But the process exits non-zero with a traceback after training has fully completed, which breaks any exit-code-based "did training succeed" check wrapped around scripts/train.py.

Reproduction

Any config where optimization.steps % checkpoints.interval == 0, run to completion on Windows, e.g.:

optimization:
  steps: 2000
checkpoints:
  interval: 250
  save_training_state: full   # or minimal — anything except off

Reproduced twice (fresh run and a resumed run) on Windows 11, Python 3.12.10, torch 2.11.0+cu128, single RTX 5090, training_mode: lora, at current main (4f89057).

Traceback

(local path prefixes abbreviated)

File "...\LTX-2\packages\ltx-trainer\scripts\train.py", line 59, in main
    trainer.train(disable_progress_bars=disable_progress_bars)
File "...\LTX-2\packages\ltx-trainer\src\ltx_trainer\trainer.py", line 323, in train
    saved_path = self._save_checkpoint()
File "...\LTX-2\packages\ltx-trainer\src\ltx_trainer\trainer.py", line 974, in _save_checkpoint
    self._save_training_state(save_dir)
File "...\LTX-2\packages\ltx-trainer\src\ltx_trainer\trainer.py", line 1039, in _save_training_state
    tmp_path.rename(state_path)
File "...\Python312\Lib\pathlib.py", line 1363, in rename
    os.rename(self, target)
FileExistsError: [WinError 183] Cannot create a file when that file already exists:
'...\\out\\my_lora\\checkpoints\\training_state_step_02000.pt.tmp' ->
'...\\out\\my_lora\\checkpoints\\training_state_step_02000.pt'

Minimal demonstration of the rename semantics

import os, tempfile
from pathlib import Path

d = Path(tempfile.mkdtemp())
target, tmp = d / "state.pt", d / "state.pt.tmp"
target.write_bytes(b"first")
tmp.write_bytes(b"second")

tmp.rename(target)         # Windows: FileExistsError [WinError 183]; POSIX: replaces
# os.replace(tmp, target)  # both platforms: atomic replace

Suggested fix

One line in _save_training_state:

-        tmp_path.rename(state_path)
+        os.replace(tmp_path, state_path)

os.replace overwrites atomically on both POSIX and Windows and changes nothing about the on-disk result.

Worth noting that the method already anticipates this exact same-step double save a few lines below — the retention list is deduped (trainer.py#L1054-L1055):

if not self._training_state_paths or self._training_state_paths[-1] != state_path:
    self._training_state_paths.append(state_path)

— only the rename was left unguarded.

(An alternative would be to skip the final _save_checkpoint() at L323 when the loop's last step already checkpointed, which would also avoid gathering the full state dict twice — but os.replace is the minimal safe change.)

Related minor note

The same double-fire also double-appends the weights path on all platforms: _save_checkpoint has no equivalent dedupe guard at trainer.py#L971 (self._checkpoint_paths.append(saved_weights_path)), so _cleanup_checkpoints counts the final checkpoint twice against keep_last_n and can retire one more old checkpoint than configured. Cosmetic and platform-independent — mentioning it here since it shares the root cause.

Happy to open a PR for the one-liner — we're running it as a local patch.