This article is Part of a series on running vLLM on AKS and walks through creating an AKS cluster with a GPU node pool, deploying vLLM onto it, and wiring up Prometheus and Grafana for visibility.
Companion pieces: Choosing the right GPU | Why your autoscaler flaps| Source Setup Summary Cloud: Azure GPU node: (1× A10, 24 GB) Image / model: serving Observability: kube-prometheus-stack (Prometheus + Grafana), KEDA, NVIDIA DCGM exporter All commands below are bash.
The steps are ordered and each one depends on the previous.
Dependency chain The build order follows one chain: model → VRAM requirement → GPU SKU → region availability → quota.
Step 0 — Prerequisites (one-time, survives resource group deletion) GPU quota.
Request through Portal → Quotas → Compute → This article: Requested in (108 = 3 nodes × 36 vCPUs, matching the autoscaler's set in step 3).
Quota is granted per-subscription and survives resource group deletion, so this step happens once, not on every rebuild.
A quota is Azure's per-subscription limit on how much of a resource (here, GPU vCPUs in a specific VM family) you're allowed to provision at once.
New subscriptions start at 0 for GPU families since it's expensive and can be abused.
You need it because without an approval, az aks nodepool add for a GPU will fail outright.
The request goes through manual Azure approval, so it has to happen before you plan to build. *Prerequisites * Existing Azure Subscription: Local tooling: Helm 3+ kubectl Bash Bash Variables to set for use through the setup Step 1 — Create Resource group Step 2 — Create AKS cluster, on a CPU system pool The GPU does not go on this pool.
Every AKS cluster requires a system node pool for cluster-critical pods (CoreDNS, metrics-server), and system pools cannot scale to zero — so a GPU placed here runs, and bills, 24/7 regardless of load.
A CPU node covers the system pods cheaply; the GPU pool created in step 3 is where scale-to-zero actually happens.
Step 3 — GPU node pool, tainted and scaled from zero What each flag does: + — scale-from-zero.
No node exists until a pod requires one, so there's no GPU spend until step
7.
The cluster autoscaler evaluates a pending pod against the pool's declared taints/labels/VM size to decide whether it would fit, then provisions a node if so — which is why the taint and label must be set on the pool at creation, not discovered later from a running node. — bounds the pool between 0 and 3 nodes. (Design decision) — blocks ordinary CPU pods from landing on the GPU node once it exists. — the label the vLLM pod's targets in step
7.
Taint, toleration, and nodeSelector do three separate jobs: the taint repels pods by default, a toleration permits a specific pod to ignore that taint, and a nodeSelector steers a pod toward a specific node.
A toleration alone doesn't guarantee placement — it only lifts the block.
The vLLM pod spec in step 7 carries both the toleration and the nodeSelector because both are required.
The above scaling sets the --node-count and --min-count to 0; this may or may not be desirable for your use case.
Keeping a node or 2 warm can help with latency, but there is cost associated with that.
Choose whichever best fits your use case.
Step 4 — Get kubeconfig for cluster communication Step 5 — NVIDIA device plugin AKS does not install this by default.
Without it, a GPU node never advertises as an allocatable resource, and any pod requesting stays indefinitely with no error.
The upstream DaemonSet only tolerates the standard taint, not the custom taint set in step 3, so it won't schedule onto the GPU node without a patch: A vLLM pod looks identical whether the device plugin is missing, mis-scheduled, or the node just hasn't scaled up yet. and checking for under distinguishes between the three.
Step 6 — Observability core: Prometheus + Grafana + KEDA NOTE: Installed before the GPU node scales up, so the stack builds on the free CPU pool.
Values file: — 6-hour retention, an 8 Gi PV on , Grafana on , Alertm