Choosing the Right GPU for Your Model — A Sizing Method, Not a Guess OK, you're a senior SRE, you've been hearing incessantly about AI models, but aren't quite sure how to determine the correct node size to host your model. - If so ... you're in the right place.
Part of a series on running vLLM on AKS.
Companion piece: How to avoid flapping.
GPU infrastructure setup — coming soon.
This piece walks through estimating GPU memory requirements from both a model's parameter count or a concurrent requests requirement.
After reading this article you will have enough knowledge to pick a GPU family with confidence.
Disclaimer: this process is a rule-of-thumb filter, not a precise calculation — the last step covers how to get exact numbers once the model is actually running.
Background: What actually consumes GPU memory AI models live in GPU memory — VRAM — and engines such as vLLM provide novel techniques for managing that memory efficiently [paper], but the model isn't the only thing consuming it.
Below is a short list of things that consume our precious VRAM: Model weights — the parameters themselves.
The big fixed cost: loaded once, never shrinks.
KV cache — working memory for in-flight requests.
Every token of every active request holds its attention keys/values here.
This is the one that determines throughput: more KV cache = more concurrent requests.
Everything else — activations (the temporary tensors of a forward pass) plus CUDA/framework overhead.
You don't calculate these by hand; vLLM measures activations with a profiling pass at start-up and prints it for our consumption.
The sizing question is really: after weights and overhead, how much is left for the KV cache — and is that enough for your traffic?
OK, lets get started Step 1 — Choose a model Guidance on which model to choose is outside the bounds of this article.
What matters here: once you have a candidate, everything below can be read off its spec sheet — you can then run this method on every model on your shortlist and eliminate the ones that don't fit your requirements.
For demonstration purposes I will use Hugging Face's Qwen2.5-7B-Instruct-AWQ model card, config card Step 2 — What to look for in the spec sheet What Name Value Qwen2.5-7B-Instruct-AWQ Parameter count model card / Number of Paremeters ~7.6 B ("7B") Quantization AWQ, 4-bit Layers 28 KV heads 4 Head dimension 3584 / 28 = 128 The first two size the weights.
The last three size the KV cache per token.
That's the whole shopping list.
Step 3 — Calculate the VRAM for the weights Rule of thumb: and many models have different precision offerings Precision Bytes/param 7.6 B params fp16 / bf16 2 ~15.2 GB int8 1 ~7.6 GB AWQ / 4-bit ~0.5 ~3.8 GB Note on Weights and quantization: the above table compares the same model, just with different quantization: Quantization is storing each weight in fewer bits than it was trained in.
Models train in 16-bit float, so every parameter costs 2 bytes (16 bits); quantizing re-encodes them as 8-bit or 4-bit integers.
So the above table shows same model, same parameter count.
Only the memory footprint for your model changes.
Every GB you don't spend on weights is a GB left for KV cache, which allows serving more concurrent requests resulting in more happy customers.
Already this helps guide decisions: The fp16 model variant (~15 GB) plus workspace would nearly fill a 24 GB GPU before serving a single request.
The AWQ variant (~5.6 GB) leaves the majority of VRAM free for the KV cache.
Same model, same GPU — wildly different serving capacity.^^ In truth, the AWQ actually consumes ~5.6 GB, not the advertise ~3.8 GB above.
This can be seen by looking at the sum of the file sizes on the repo's Files and versions tab For this model that's exactly two files: model-00001-of-00002.safetensors (~4.0 GB) and model-00002-of-00002.safetensors (~1.6 GB).
These are the weight tensors themselves — the files vLLM downloads and loads into VRAM at startup — so their combined size is the weights fo