#738·nanoGPT

GPT.generate() crashes on temperature=0 instead of doing greedy decoding

Author: NikhilSKashyapCreated Jun 17, 2026Updated Jun 17, 2026

Description

### Summary
`GPT.generate()` divides the logits by `temperature` unconditionally. With
`temperature=0` — the standard convention for deterministic / greedy decoding
(OpenAI, Hugging Face, llama.cpp all treat 0 as "argmax") — this is a division
by zero: the logits become `inf`/`nan`, softmax yields `nan` probabilities, and
`torch.multinomial` raises. So the most natural way to ask for deterministic
output crashes instead of working.

### Steps to reproduce
```python
import torch
from model import GPT, GPTConfig

cfg = GPTConfig(block_size=16, vocab_size=32, n_layer=1, n_head=1,
                n_embd=16, dropout=0.0, bias=False)
m = GPT(cfg).eval()
idx = torch.zeros((1, 1), dtype=torch.long)
m.generate(idx, max_new_tokens=5, temperature=0.0)   # boom

Actual behavior

RuntimeError: probability tensor contains either `inf`, `nan` or element < 0

Expected behavior

temperature=0 should perform greedy decoding (pick argmax of the logits), not crash — consistent with every other major sampling implementation.

Root cause

model.py, in GPT.generate():

logits = logits[:, -1, :] / temperature   # temperature == 0 -> inf/nan

Suggested fix

Short-circuit temperature == 0 to greedy decoding before the division:

logits = logits[:, -1, :]
if temperature == 0.0:
    idx_next = torch.argmax(logits, dim=-1, keepdim=True)
else:
    logits = logits / temperature
    # ... existing top-k + multinomial path