[Bug] scheduler: `assert scheduled_seqs` crashes engine when decode runs out of KV-cache blocks

Description

Scheduler.schedule() crashes with AssertionError: scheduled_seqs in the decode phase whenever the KV cache is exhausted and preemption cannot make room — a legitimate backpressure condition that the surrounding code is explicitly written to handle.

# nanovllm/engine/scheduler.py, decode phase
while self.running and len(scheduled_seqs) < self.max_num_seqs:
    seq = self.running.popleft()
    while not self.block_manager.can_append(seq):
        if self.running:
            self.preempt(self.running.pop())
        else:
            self.preempt(seq)
            break
    else:
        ...
        scheduled_seqs.append(seq)
assert scheduled_seqs          # <-- crashes here
self.running.extendleft(reversed(scheduled_seqs))

How to reproduce

Minimal repro, no model/weights needed (drives Scheduler + BlockManager directly with a 2-block KV cache; single sequence whose prompt fills the whole cache):

Minimal runnable repro (no torch/triton/model weights; drives the scheduler directly):

import importlib.util, sys, types
from types import SimpleNamespace
ROOT = "<repo-root>"
def load(name, path):
    spec = importlib.util.spec_from_file_location(name, path)
    m = importlib.util.module_from_spec(spec); sys.modules[name] = m
    spec.loader.exec_module(m); return m
pkg = types.ModuleType("nanovllm"); pkg.__path__ = [f"{ROOT}/nanovllm"]
sys.modules["nanovllm"] = pkg
eng = types.ModuleType("nanovllm.engine"); eng.__path__ = [f"{ROOT}/nanovllm/engine"]
sys.modules["nanovllm.engine"] = eng
cfgstub = types.ModuleType("nanovllm.config"); cfgstub.Config = object
sys.modules["nanovllm.config"] = cfgstub
seq_m = load("nanovllm.engine.sequence", f"{ROOT}/nanovllm/engine/sequence.py")
load("nanovllm.engine.block_manager", f"{ROOT}/nanovllm/engine/block_manager.py")
sch_m = load("nanovllm.engine.scheduler", f"{ROOT}/nanovllm/engine/scheduler.py")

S, NB = 4, 2  # block_size=4, cache holds only 2 blocks = 8 tokens
cfg = SimpleNamespace(max_num_seqs=8, max_num_batched_tokens=64, eos=-1,
                      kvcache_block_size=S, num_kvcache_blocks=NB)
Sequence = seq_m.Sequence; Sequence.block_size = S
sch = sch_m.Scheduler(cfg)
sch.add(Sequence(list(range(8)), SimpleNamespace(temperature=0.0, max_tokens=100, ignore_eos=False)))
for step in range(5):
    seqs, is_prefill = sch.schedule()
    if not seqs:
        print(f"step {step}: empty batch (is_prefill={is_prefill})")
        break
    sch.postprocess(seqs, [1000 + step] * len(seqs), is_prefill)  # dummy non-EOS output
    s = seqs[0]
    print(f"step {step}: pre={int(is_prefill)} len={s.num_tokens} cached={s.num_cached_tokens} blocks={len(s.block_table)}")

Output (crash on step 1):

step 0: pre=1 len=9 cached=8 blocks=2
Traceback (most recent call last):
  .../nanovllm/engine/scheduler.py, line 74, in schedule
    assert scheduled_seqs
AssertionError: assert scheduled_seqs

The real-model equivalent: set max_tokens large enough that one sequence's prompt+generation exceeds num_kvcache_blocks * kvcache_block_size; the engine crashes the same way mid-generation.

Analysis

The decode loop is entered exactly when the prefill phase scheduled nothing (if scheduled_seqs: return ...). The self.preempt(seq); break branch is designed for this situation: the last remaining running sequence is moved back to waiting (its blocks deallocated) so the next step can retry. But the trailing assert scheduled_seqs turns that graceful path into a hard process crash.

Two consequences:

1. With asserts on (default): process dies with AssertionError on KV-cache exhaustion (e.g. single long-running sequence whose total length exceeds the KV cache, or a cache too small to fit one full sequence).
2. With python -O (asserts stripped): schedule() returns ([], False). LLMEngine.step() then returns num_tokens = 0, and LLMEngine.generate()'s while not self.is_finished() loop spins forever with empty batches, because the preempted sequence stays in waiting (is_finished() is never True) and the re-prefill also cannot fit — a silent infinite busy-loop instead of a crash.

Either way, there is no way for the engine to observe "decode had nothing to run" — the empty-batch signal the caller needs (idle / OOM backpressure) is destroyed.

Suggested fix

• In schedule(): drop assert scheduled_seqs and return the (possibly empty) decode batch — the return scheduled_seqs, False already carries the "nothing to run" signal.
• In the engine: treat a decode step that scheduled zero sequences as a stall condition (stop/back-off or raise a clear "KV cache out of memory" error) instead of silently re-entering the loop. Note a preempted-then-re-prefilled sequence can still fail to fit, so this is a real engine-level state, not a scheduler invariant.

Environment

• Branch: main (present on upstream @main, scheduler.py:74), also HEAD chunked-prefill-refactor merge (#218)
• Python 3.x, no GPU needed to reproduce

Related but different: #217 fixed a separate AssertionError (postprocess re-prefill early-exit); #240 discusses a different can_append/may_append question.