Harden downloaded dataset shard cache in prepare.py
Author: vincentayorindeCreated Mar 12, 2026Updated Apr 16, 2026
Problem
Right now prepare.py downloads dataset shard files and reuses the local cached copies later, but those cached shard files are not verified before reuse.
That is usually fine for normal local runs, but it becomes risky on shared machines, copied cache directories, or if a cached file becomes corrupted or is tampered with.
Fix
A small fix would be to add simple integrity verification for cached dataset shard files before reuse, and redownload them if they are missing, incomplete, or fail verification.
Source: karpathy/autoresearch