Originally published on hexisteme notes.
I run a small fleet of coding agents on one machine.
Every thread ends up in a log, and a measurement pipeline reads those logs into a database, attributing each turn to the model that produced it.
After a few thousand threads I had the table people keep asking for: seven model-versions, five behavioural metrics, real workload rather than a benchmark.
Then I printed one cross-tab I had been skipping, and most of that table stopped meaning what I thought it meant.
The cross-tab was role × model.
In this fleet a model runs in one of two roles.
It is either the long interactive main thread I drive by hand, or a short one-shot sub-agent that a main thread spawns, runs once, and discards.
Same model.
Same weights.
Two completely different jobs.
Role turned out to move the numbers by up to 135x, and the role mix is wildly different for each model.
Those two facts together are enough to make a pooled comparison manufacture a large gap that exists in neither stratum.
A note on labels: model names are replaced with letters on purpose.
The point of this note is that these numbers are not a model ranking, and printing the names invites exactly that misreading.
Every figure is real, from one snapshot of one fleet.
The role gap is an order-of-magnitude thing Median output tokens per thread, same model, split by role: model main n main median sub-agent n sub-agent median main ÷ sub A 200 478,238 2,761 6,212 77x C 88 427,838 156 8,784 49x D 11 185,534 122 1,371 135x G 21 128,415 18 11,358 11x B 20 16,432 1,338 14,157 1.2x F 15 4,194 6 51,562 0.08x The behavioural metrics are worse than lopsided — in one stratum they are flat: metric main-thread medians sub-agent medians same-file re-edit rate 0.40 / 0.44 / 0.50 / 0.53 (four models) exactly 0 for six of seven error-recovery sequences 1 / 2 / 2 / 2 (four models) exactly 0 for six of seven validation runs 0 for six of seven 0 for all seven The "(four models)" qualifier is load-bearing, so let me not hide behind it: the other three model-versions sit at a median of 0 in main too — two genuinely, one because its main cell holds 2 rows and is marked not comparable.
Main is where between-model signal can live, not where it always does.
Still, the pattern is not a subtle covariate.
A main thread iterates: read, edit, re-edit the same file, hit a failure, recover, run a check.
A sub-agent is fire-and-forget — it does its one job and exits, so it rarely touches the same file twice and rarely has a failure to recover from.
The metric is structurally near-zero there.
Which means: for re-edit rate and recovery count the sub-agent stratum carries no between-model signal at all — the median is a constant.
All the signal lives in the main-thread stratum, 7% of my rows.
The mix is different for every model Here is the cross-tab I should have printed on day one: model main rows sub-agent rows main share E 2 132 1.5% B 20 1,338 1.5% A 200 2,761 6.8% D 11 122 8.3% C 88 156 36.1% G 21 18 53.8% F 15 6 71.4% Across the whole attributed corpus, 4,533 of 4,890 rows — 92.7% — are sub-agent rows.
So a pooled number is mostly a description of sub-agents.
But how mostly ranges from 1.5% main to 71.4% main, a 48-fold spread in composition.
The reason is not random sampling — it is the delegation policy.
My orchestration rules send mechanical fan-out work to cheaper tiers, so those models accumulate sub-agent rows by the thousand; models I drive by hand accumulate main rows.
The edge counts show it directly: model A spawned 1,788 sub-agents that were also A and 730 that were B; model C spawned 393 that were B.
Role is assigned by the same policy that assigns the model: the confounder is baked into the architecture, not introduced by chance.
Anything with an orchestration layer has this shape — retry tiers, canary vs steady-state traffic, batch vs interactive queues, free vs paid users.
The router picks both which variant handles a request and what kind of request it is.
The pooled