The Flag We Tuned Around Got Deleted

2026年9月8日2 次浏览来源:Dev.to阅读原文

The single most important llama.cpp flag for my dual Tesla P40 setup was .

It split every layer's tensors across both GPUs and it was worth nearly double the throughput of the alternative: 12-14 tokens/sec against about 7 for layer split.

Every stack I built was tuned around it.

In July 2026, upstream llama.cpp deleted it.

Not deprecated.

Deleted.

This is the story of a performance rule that died twice, and what replaced the throughput it took with it.

It's the longest arc in my notes, and it runs in five acts.

Act 1: Row wins February

2026.

My first serious model, a 72B, started at 2.7 to 3.6 tokens/sec in what my notes politely call "poor configuration." Working up the offload ladder to full GPU residency and switching to row split produced the first real daily driver: about 10.3 tokens/sec generation, 60 tokens/sec prompt processing.

The standing rule crystallized: row split, 12-14 tokens/sec.

Layer split, about

7.

And a rumor from a vendor blog said a newer "graph" split mode was worth another 30-40%.

Act 2: The regression incident March

2026.

I rebuilt the same model on a "modernized" fork, and changed four variables at once: split mode, SIMD compile flags, kernel selection method, and compression settings.

Prompt processing collapsed from 153 tokens/sec to

29.

Five times slower, and with four simultaneous changes, nothing was attributable.

The cleanup A/B on March 4 isolated everything.

Same model, same prompt, one variable at a time: Original binary, row split: 60 tokens/sec prompt, 10.3 generation.

Works.

New fork, layer split: half the speed.

New fork, graph split: CUDA crash. .

The "40% faster" graph mode does not run on Pascal at all.

That's the difference between a community claim and a measurement on your own hardware: one of them can crash.

The verdict written that day: the original binary is optimal for this hardware, do not switch.

And a rule was born that I now treat as non-negotiable: one variable per change.

That rule was paid for in lost throughput and a wasted week.

Act 3: The fast mode becomes the wrong mode May

2026.

Gemma 4 arrived, and its architecture uses shared KV layers implemented as tensor views.

Those crash row split on multi-GPU, a hard assert deep in the CUDA backend, known upstream issue.

Every Gemma stack I run is layer split by necessity, knowingly paying the throughput cost.

The Qwen architectures have no such bug and kept row.

So split mode is not a performance dial.

It's also a correctness knob, and the answer is per-architecture. "Row is faster" was true and useless without the condition attached.

Act 4: Upstream deletes row July 6,

  1. llama.cpp removed entirely.
    Any stack built from a newer clone has layer split as its only multi-GPU option.
    The flag I had organized my fleet around no longer exists in the binaries I build.
    Row died twice, in two lineages: first the fork replaced it with a mode that crashes Pascal, then upstream removed it outright.
    Act 5: The win comes back from somewhere else Here's the part that justified the whole ordeal.
    On the new stack, layer split alone measured 8.46 tokens/sec single-stream, right where the old "layer is about 7" rule predicted.
    But two features that didn't exist in my February binaries changed the math: (four concurrent slots): 8.46 / 12.8 / 15.0 tokens/sec at 1, 2, and 4 slots.
    Aggregate throughput nearly doubles before per-slot latency degrades.
    MTP speculative decoding: 8.46 to about 13.3 tokens/sec single-stream, a 57% lift, acceptance rates 0.38 to 0.63, output correctness verified.
    Net result: the fleet ended up faster than the row-split era without row split.
    The recovery didn't come from finding a replacement flag.
    It came from features orthogonal to the one I lost.
    What this taught me Date-stamp every performance claim and name the binary it was measured on.
    A tuning rule is a fact about a specific artifact at a specific moment, not a law of the hardware.
    And when the flag you tuned around disappears, re-measure before assuming regression.
    The replacement win may live somewhere you weren't looking.
    Mine did, twice over.
分享