This article provides a step by step comparison of three Gemma 4 deployments on a single AWS hosted GPU enabled system.
A suite of Python MCP tools is built to simplify management of each deployment, and one benchmark harness is shared across all three so that the runtime is the only variable. https://github.com/xbill9/gemma4-dev The whole exercise cost under three dollars, and that is the part worth keeping.
Nineteen instances and about four and a half instance-hours bought three serving sweeps, nine timed boots and a handful of A/B restarts.
It also bought five wrong claims, each caught by measuring instead of reasoning.
On hardware where a run is expensive, the cheapest of those five would have shipped with a caveat attached.
What is this project trying to Do?
Three rigs in this monorepo serve on an AWS G5g instance.
One runs vLLM, one runs a pure JAX port, one runs PyTorch with transformers.
The hardware is identical and only the runtime slot moves, so this should be the cleanest A/B available.
For months it was not, because each rig measured itself with its own harness and quoted its own number.
Three harnesses computing three statistics is not a comparison.
Prerequisites An AWS account with G-family quota in .
Each is 8 vCPU, so 16 vCPU of spot quota runs two at once.
A subnet, a security group opening TCP 8000, and an instance profile carrying plus read on the Hugging Face token secret.
A Hugging Face token in Secrets Manager.
It is fetched at boot into a root-only and never appears in user data. and the standard credential chain.
No AWS CLI shell-outs, no inbound SSH rule, and no private key anywhere in the flow.
AWS EC2 G5g Instance — 8 vCPU, 16 GiB host Host CPU AWS Graviton2, aarch64 GPU 1x NVIDIA T4G, Turing, SM 7.5 GPU memory 15,360 MiB per ; AWS lists 16,384 nominal G5g is the only family AWS ships that puts an NVIDIA GPU behind a Graviton host, which makes it the only place to get aarch64 and compute capability 7.5 together.
Gemma 4 E2B is the reference instruction-tuned release.
It is 2B effective from about 5B total, and the split matters here: most of what is resident is a per-layer-embedding table that decode reads as a gather and never streams through a matmul.
The dense checkpoint fits. 9.5 GiB of float16 weights go into 15,360 MiB of device memory with room for the KV cache, which at roughly 18 KiB per token is tens of megabytes at this context and never the binding constraint.
The Three Runtimes runtime engine how it serves vLLM v0.27.2rc0, built from source for continuous batching, paged KV, prefix caching JAX this project's own port hand-written KV ring with a bucket ladder PyTorch + transformers , one request at a time Turing has no bfloat16 datapath, so all three run float16.
It has no fp8 either, which rules out the KV-cache tricks that work on newer parts.
Check the Quotas That is the constraint behind every launch below: two rigs in parallel, and no more.
The Sweep Could Not See vLLM The sweep script read its throughput figure straight out of the response body: is a field our own servers invent. vLLM does not emit it, and neither does anything else, so the harness could not be pointed at the vLLM rig at all.
The three-way comparison had never actually been run.
Re-running a rig does not fix that.
Only a common statistic does.
One Statistic, Three Servers Every OpenAI-compatible server streams, so the portable measurement is the gap between tokens on the wire.
The path uses 's exact TPOT definition, , so a number from this harness is directly comparable to that tool's published figures. probes the endpoint once and picks where the server emits its own gauge, where it does not.
Is the Calibration Transferable?
No, and that is worth a measurement rather than an assumption.
Running measures each rig's offset between the two statistics. rig server gauge client stream stream/gauge JAX 12.962 12.687 0.9799 PyTorch 10.814 10.243 0.9543 Two percent against 4.6 percent, on the same day and the same instance