#5713·burn

pytorch-reader: LegacySource reopens the checkpoint by path on every read

Author: antimoraCreated Sep 17, 2026Updated Sep 17, 2026
Labelsbugstore

Summary

LegacySource::read_storage calls File::open(&self.path) on every read. ZipSource (after the fix for #5711) holds its handles from open until drop and never reopens by path. The two containers therefore behave differently when the file goes away: a ZIP reader keeps reading the file it opened, a legacy reader fails with NotFound (test_os_errors_keep_their_kind pins this).

The reopen is also a correctness hazard under atomic replacement. A training loop that renames the next epoch's checkpoint over the path while an evaluator holds a reader makes that reader read the new file at the old offsets. When the layout matches (same model), the stored_numel check passes and the tensor comes back as a torn mix of old and new bytes, with no error.

Fix sketch

LegacySource { file: File, state } and new(file); load_legacy already opens the file on its first line and can hand it over. read_storage becomes two positional reads (the i64 count at offset, then len bytes at offset + 8 + start) through the read_exact_at helper that the #5711 fix adds to storage.rs. finish already validates end <= file_len, so the allocation is bounded by the same argument ZipSource::read_stored uses. About 15 lines net.

Caveat

test_os_errors_keep_their_kind uses the legacy reopen to get an OS-level error at read time; it exists to check that build_tensor preserves OS error kinds instead of folding them into InvalidData. That path needs a different trigger once the file is held open.