#590·nanochat

[错误] scripts/chat_sft.py 在小设备批量大小(≤ 8)下,从步骤 00001 开始产生损失:nan,原因是完全掩码的微批量

作者: suraj-self创建于 2026年3月5日更新于 2026年8月26日
标签code robustness

Description: When running Supervised Fine-Tuning (SFT) on hardware with limited VRAM (e.g., 6GB/8GB GPUs), setting a small device-batch-size (e.g., 4 or 8) consistently leads to NaN loss values starting from Step 1. This appears to be a regression or unhandled edge case in how the SFT trainer handles micro-batches that contain no active training targets.

内容来源: karpathy/nanochat