nanochat · Issues· 120 open

SFT inherits pre-batch-scaling learning rates from base checkpoint

Multiple causal-leakage architectures can collapse NanoChat val_bpb to 0.0018

TaskSequence in tasks/common.py is unused

"Peak memory usage" reports 0.00MiB on CPU and MPS

[Bug] scripts/chat_sft.py produces loss: nan from step 00001 on small device-batch-size (≤8) due to fully-masked micro-batches

chat_rl: the sampled assistant_end token is masked out, so stopping is never reinforced

SDPA fallback: sliding window doesn't reduce memory (full mask built regardless of window size)

RTX 3050TI Not detecting m.get_device()

Use Official PyTorch FA3 Builds

FA3 loader reports success on Blackwell (sm_120) then dies at first kernel launch

Token smearing does not correctly handle chunked prefill / chunk inference (first token of a chunk misses its cross-chunk predecessor

Add support for TPU?

Add support for OpenCL (e.g., for the integrated AMD GPU)?

TODO: hash for d34 model

base_eval.py: hellaswag gets progressively slower and leaks memory on small models (Mac Studio)