This article provides a step by step comparison of three Gemma 4 deployments on a single AWS hosted GPU enabled system.
A suite of Python MCP tools is built to simplify management of each deployment, and one benchmark harness is shared across all three so that the runtime is the only variable. https://github.com/xbill9/gemma4-dev The whole exercise cost under three dollars, and that is the part worth keeping.
Nineteen instances and about four and a half instance-hours bought three serving sweeps, nine timed boots and a handful of A/B restarts.
It also bought five wrong claims, each caught by measuring instead of reasoning.
On hardware where a run is expensive, the cheapest of those five would have shipped with a caveat attached.
What is this project trying to Do?
Three rigs in this monorepo serve on an AWS G5g instance.
One runs vLLM, one runs a pure JAX port, one runs PyTorch with transformers.
The hardware is identical and only the runtime slot moves, so this should be the cleanest A/B available.
For months it was not, because each rig measured itself with its own harness and quoted its own number.
Three harnesses computing three statistics is not a comparison.
Prerequisites An AWS account with G-family quota in .
Each is 8 vCPU, so 16 vCPU of spot quota runs two at once.
A subnet, a security group opening TCP 8000, and an instance profile carrying plus read on the Hugging Face token secret.
A Hugging Face token in Secrets Manager.
It is fetched at boot into a root-only and never appears in user data. and the standard credential chain.
No AWS CLI shell-outs, no inbound SSH rule, and no private key anywhere in the flow.
AWS EC2 G5g Instance — 8 vCPU, 16 GiB host Host CPU AWS Graviton2, aarch64 GPU 1x NVIDIA T4G, Turing, SM 7.5 GPU memory 15,360 MiB per ; AWS lists 16,384 nominal G5g is the only family AWS ships that puts an NVIDIA GPU behind a Graviton host, which makes it the only place to get aarch64 and compute capability 7.5 together.
Gemma 4 E2B is the reference instruction-tuned release.
It is 2B effective from about 5B total, and the split matters here: most of what is resident is a per-layer-embedding table that decode reads as a gather and never streams through a matmul.
The dense checkpoint fits. 9.5 GiB of float16 weights go into 15,360 MiB of device memory with room for the KV cache, which at roughly 18 KiB per token is tens of megabytes at this context and never the binding constraint.
The Three Runtimes runtime engine how it serves vLLM v0.27.2rc0, built from source for continuous batching, paged KV, prefix caching JAX this project's own port hand-written KV ring with a bucket ladder PyTorch + transformers , one request at a time Turing has no bfloat16 datapath, so all three run float16.
It has no fp8 either, which rules out the KV-cache tricks that work on newer parts.
Check the Quotas That is the constraint behind every launch below: two rigs in parallel, and no more.
The Sweep Could Not See vLLM The sweep script read its throughput figure straight out of the response body: is a field our own servers invent. vLLM does not emit it, and neither does anything else, so the harness could not be pointed at the vLLM rig at all.
The three-way comparison had never actually been run.
Re-running a rig does not fix that.
Only a common statistic does.
One Statistic, Three Servers Every OpenAI-compatible server streams, so the portable measurement is the gap between tokens on the wire.
The path uses 's exact TPOT definition, , so a number from this harness is directly comparable to that tool's published figures. probes the endpoint once and picks where the server emits its own gauge, where it does not.
Is the Calibration Transferable?
No, and that is worth a measurement rather than an assumption.
Running measures each rig's offset between the two statistics. rig server gauge client stream stream/gauge JAX 12.962 12.687 0.9799 PyTorch 10.814 10.243 0.9543 Two percent against 4.6 percent, on the same day and the same instance shape.
Borrowing one rig's ratio to convert the other's number would inject a 2.6 percent error into a comparison whose smallest interesting gap is 24 percent.
The cross-rig table below is therefore built from throughout.
A Gauge Rounded to One Decimal The JAX server emitted its decode gauge with one decimal place: At about 13 tok/s, one decimal is 0.78 percent resolution.
Every sweep that rig had produced showed all three repeats of a cell as byte-identical: 12.8, 12.8, 12.8.
That is not reproducibility, it is the measurement floor.
The rig had been used to argue about two percent effects it could not resolve.
Two characters fixed it.
The first run afterwards reads 12.962, where before it would have said 13.0.
Launch the Instance Capacity for the whole G5g family was exhausted across all four availability zones several times, so the launcher cycles them with a sixty second backoff.
The walkthrough from here follows the PyTorch rig on ; the other two run the same steps against their own instances.
All three landed in within hours of each other.
Note that AWS names the other zones as available in every one of those errors — that text describes on-demand capacity and says nothing about spot.
Watch the Install Cloud-init installs the runtime and then backgrounds itself, so the progress tool reports cloud-init's own state as well as the install log.
A dead bootstrap and a slow one must not render identically.
This is a wheel install, not a build.
Across the three timed boots the install finishes a median 113.55 s after launch, against the hours the vLLM rig needs for a from-source build.
Verify the GPU A config flag being accepted proves nothing, so the probe runs a real matmul on the device.
The DLAMI's torch carries .
Upstream PyPI aarch64 wheels do not, so a on this box would serve on CPU without saying so.
Deploy the Server The payload is the rig's own source, shipped over SSM as a gzipped tarball because user data caps at 16 KiB.
Verify the Installation A non-empty reply is not evidence of health.
One sibling was once measured answering , so the check reads the server's own degenerate-response counter either side of its probe, and compares the served build id against the local payload.
Run the Sweep The same command runs against all three rigs.
Only the endpoint changes.
Cells that cannot exist on the hardware are recorded as rather than dropped.
An absent cell is indistinguishable from an untried one, which is how a sweep overstates its own coverage.
Decode at Concurrency One runtime decode tok/s % of ceiling vs PyTorch cells 🥇 vLLM v0.27.2rc0 32.53 53.0% 3.18x 12/12 🥈 JAX 12.69 20.7% 1.24x 10/12 🥉 PyTorch + transformers 10.24 16.7% 1.00x 10/12 How Close Is That to the Hardware?
The ceiling is arithmetic, not a measurement.
E2B streams 4.514 GB of weights per decode step against a measured 277 GB/s, giving 16.30 ms per step and 61.4 tok/s.
The PLE table is excluded from that figure because it is a gather and never a matmul.
Quantising it from 9.257 GB to 5.752 GB moved decode by 0.00 tok/s across three cells, which is what confirms it never streams.
All three runtimes sit far below the ceiling, so none of them is bandwidth-bound at batch one.
The PyTorch profile shows why: about 5,650 kernel launches per step at one to three microseconds each, on a chip whose launch overhead is five to ten.
The Number I Did Not Expect Time to first token was the result of the run, right up until it was checked. input tok vLLM JAX PyTorch 92 103 ms 225 ms 164 ms 1,259 118 ms 1,615 ms 657 ms 3,746 178 ms 5,352 ms 2,339 ms A 30x advantage, far larger than the 3.2x on decode.
It is also impossible.
Prefill at 3,746 tokens is roughly 14 TFLOP against a T4G's realistic 20 to 30 TFLOP/s, which is 460 ms at best. vLLM's row says 178 ms.
It did not. vLLM ships , and its own metrics say so: A 94.7 percent hit rate. vLLM genuinely prefilled 5.3 percent of the tokens it was sent, because the harness reused one prompt for a cell's warm-up and all thr