The single most important llama.cpp flag for my dual Tesla P40 setup was .
It split every layer's tensors across both GPUs and it was worth nearly double the throughput of the alternative: 12-14 tokens/sec against about 7 for layer split.
Every stack I built was tuned around it.
In July 2026, upstream llama.cpp deleted it.
Not deprecated.
Deleted.
This is the story of a performance rule that died twice, and what replaced the throughput it took with it.
It's the longest arc in my notes, and it runs in five acts.
Act 1: Row wins February
2026.
My first serious model, a 72B, started at 2.7 to 3.6 tokens/sec in what my notes politely call "poor configuration." Working up the offload ladder to full GPU residency and switching to row split produced the first real daily driver: about 10.3 tokens/sec generation, 60 tokens/sec prompt processing.
The standing rule crystallized: row split, 12-14 tokens/sec.
Layer split, about
7.
And a rumor from a vendor blog said a newer "graph" split mode was worth another 30-40%.
Act 2: The regression incident March
2026.
I rebuilt the same model on a "modernized" fork, and changed four variables at once: split mode, SIMD compile flags, kernel selection method, and compression settings.
Prompt processing collapsed from 153 tokens/sec to
29.
Five times slower, and with four simultaneous changes, nothing was attributable.
The cleanup A/B on March 4 isolated everything.
Same model, same prompt, one variable at a time: Original binary, row split: 60 tokens/sec prompt, 10.3 generation.
Works.
New fork, layer split: half the speed.
New fork, graph split: CUDA crash. .
The "40% faster" graph mode does not run on Pascal at all.
That's the difference between a community claim and a measurement on your own hardware: one of them can crash.
The verdict written that day: the original binary is optimal for this hardware, do not switch.
And a rule was born that I now treat as non-negotiable: one variable per change.
That rule was paid for in lost throughput and a wasted week.
Act 3: The fast mode becomes the wrong mode May
2026.
Gemma 4 arrived, and its architecture uses shared KV layers implemented as tensor views.
Those crash row split on multi-GPU, a hard assert deep in the CUDA backend, known upstream issue.
Every Gemma stack I run is layer split by necessity, knowingly paying the throughput cost.
The Qwen architectures have no such bug and kept row.
So split mode is not a performance dial.
It's also a correctness knob, and the answer is per-architecture. "Row is faster" was true and useless without the condition attached.
Act 4: Upstream deletes row July 6,
- llama.cpp removed entirely.
Any stack built from a newer clone has layer split as its only multi-GPU option.
The flag I had organized my fleet around no longer exists in the binaries I build.
Row died twice, in two lineages: first the fork replaced it with a mode that crashes Pascal, then upstream removed it outright.
Act 5: The win comes back from somewhere else Here's the part that justified the whole ordeal.
On the new stack, layer split alone measured 8.46 tokens/sec single-stream, right where the old "layer is about 7" rule predicted.
But two features that didn't exist in my February binaries changed the math: (four concurrent slots): 8.46 / 12.8 / 15.0 tokens/sec at 1, 2, and 4 slots.
Aggregate throughput nearly doubles before per-slot latency degrades.
MTP speculative decoding: 8.46 to about 13.3 tokens/sec single-stream, a 57% lift, acceptance rates 0.38 to 0.63, output correctness verified.
Net result: the fleet ended up faster than the row-split era without row split.
The recovery didn't come from finding a replacement flag.
It came from features orthogonal to the one I lost.
What this taught me Date-stamp every performance claim and name the binary it was measured on.
A tuning rule is a fact about a specific artifact at a specific moment, not a law of the hardware.
And when the flag you tuned around disappears, re-measure before assuming regression.
The replacement win may live somewhere you weren't looking.
Mine did, twice over.