A real-world war story of chasing an Apigee X evaluation organization from "Server Error" to "Hello, Guest!" — the hard way, with , , and a lot of patience.
TL;DR I spent hours debugging a persistent on a fresh Apigee X evaluation organization.
The Cloud Console showed every provisioning step green.
The Apigee Management API confirmed the organization and runtime instance were both .
And yet every single request — even to a brand-new proxy — came back with a
502.
The eventual root cause: the Apigee provisioning wizard built the proxy-forwarding instance template without a service account attached, and left a required metadata key blank.
Without a service account, the VMs couldnt authenticate to Cloud Storage to fetch their real startup script.
Without , the fallback script had nothing to forward traffic to.
The fix was to clone the broken instance template with the missing service account, roll the managed instance group onto it, then manually patch the metadata and re-run the startup script on each instance.
If youre staring at a 502 on a "successfully" provisioned Apigee eval org, this post is for you.
Background: how I got here I was setting up Apigee purely to practice — nothing fancy, no production intent, just a personal free-trial Google Cloud project to learn the platform.
Along the way I made a very ordinary mistake: while creating a second project to experiment with, it ended up parented under "No organization" instead of my actual Cloud Identity org, because I had several Console tabs open switching between projects mid-setup.
Confused by the mismatch, I deleted that second project.
Shortly after, my original project’s Apigee evaluation setup started throwing a resource-locked error on the "Apigee evaluation organization" step, with a message like: That project’s setup never fully completed — the "Access routing" step permanently showed "Not configured." At that point, rather than trying to excavate a half-provisioned org with a stuck long-running operation, I made the call to shut down both projects and start clean in a brand-new project.
This is a legitimate, low-cost strategy for Apigee eval orgs specifically — they’re free, disposable, and tied to your Cloud Billing account (not the project) for trial credits and the 60-day clock, so nuking a broken project costs you nothing.
Lesson #1: if youre mid-setup and juggling multiple projects, do it in separate browser profiles or windows, not tabs.
The Cloud Console’s single-page-app state can leak between tabs in confusing ways (I saw one project’s setup page briefly display another project’s Apigee org name — almost certainly a UI caching artifact, not a real resource collision).
The rebuild: a promising start, then déjà vu With a fresh project, I walked through Apigee → Try Apigee for free → Set up Apigee Evaluation step by step, in a single tab, waiting for each of the four steps to go green before touching anything else: Enable APIs ✅ Networking ✅ Apigee evaluation organization Access routing Step 3 immediately failed with a generic "Something went wrong" and offered a "Try Again" button.
Naturally, my first thought was: here we go again.
But clicking "Try Again" resolved it within about nine minutes — which felt suspicious.
Org creation is supposed to be a slow, heavyweight operation.
Why would a failed attempt succeed almost instantly on retry?
Why the quick retry wasnt fishy It turns out this is documented, known behavior.
Google’s own Apigee installation troubleshooting guide calls out this exact failure mode: If Apigee returns a 403 when you first try to create the new organization, it could mean that one or more of your APIs have not been enabled...
If you enabled this API recently, wait a few minutes for the action to propagate to our systems and retry.
Org creation is an asynchronous long-running operation (LRO) layered on top of API enablement from the prior step.
There’s a small window where the APIs report as enabled but havent fully propagated through Google’s internal systems yet.
The wizard’s org-creation call hits that window, gets rejected, and shows "Something went wrong" — but the expensive networking/VPC-peering work from Step 2 already succeeded, so once the propagation catches up, the retry only has to do the comparatively cheap part (create the org record) and finishes fast.
Dont trust a UI checkmark on faith, though — verify independently.
From Cloud Shell: Look for in the response.
This hits the Apigee Management API directly, bypassing the Console entirely — the actual ground truth, not a rendered checkmark.
Step 4 (Access routing) subsequently completed cleanly, and the wizard showed the triumphant "Congratulations!
Your Apigee organization is ready" message with a "Launch" button pointing at a test URL.
The real bug: green everywhere, 502 everywhere I clicked Launch.
I waited the requisite 15–20 minutes for what I assumed was normal load-balancer propagation lag, then tried again.
Still
502.
I deployed a fresh test proxy just to rule out something proxy-specific.
Still
502.
This is the point where "just wait longer" stops being a reasonable answer and it’s time to actually verify each layer of the stack independently, from the top down.
Layer 1: Is the request even reaching Apigee’s infrastructure?
The verbose output showed a clean TLS handshake with a valid, freshly-issued Google-managed certificate, HTTP/2 negotiated, and a response that came back structured exactly like Google’s generic Global Front End (GFE) "backend unhealthy" page — not a DNS failure, not a connection refusal, not an SSL error.
This told me the request was reaching Google’s load balancer correctly; the problem was somewhere behind it.
Layer 2: Is the Apigee org/instance actually healthy?
This came back with a real internal host/port and service attachment.
So the actual Apigee runtime — the expensive, slow-to-provision Message Processor infrastructure — was completely fine.
Layer 3: Are environment, deployment, and hostname binding all correct?
Three more calls, all clean: Environment attached, proxy deployed, hostname bound and .
Every configuration object in the Apigee control plane was correct.
This ruled out the entire Apigee-managed side of the stack.
The problem had to be in the one piece of infrastructure that Apigee’s "Access routing" step provisions but doesnt directly manage day-to-day: the Compute Engine load balancer and its backing instance group.
Layer 4: The load balancer’s backend health Apigee X’s Access Routing step builds a small Compute Engine managed instance group (MIG) of lightweight forwarding VMs that sit between the external HTTPS Load Balancer and the actual Apigee runtime instance (which lives on a private/internal IP).
Checking this directly: There it was.
Zero of two backend instances were passing their health check.
That’s a 502 by design — a GFE-class load balancer will never forward traffic to a backend it considers unhealthy, no matter how long you wait.
Root-causing the unhealthy backend Ruling out the obvious suspect: firewall rules The most common cause of "backend permanently unhealthy" on a fresh GCP load balancer is a missing firewall rule allowing Google’s health-check probe ranges ( and ) through to the target instances.
I checked: The rule existed, with the correct source ranges, correct port (443), and a matching target tag () — and a quick confirmed the actual VMs carried that exact tag.
Firewall theory eliminated.
Reading the actual boot log With the network layer cleared, the next move was to read what the instance itself thought happened during boot: Buried in there was the actual smoking gun: The VM had no service account attached at all.
Without one, it can’t mint an OAuth token, so it can’t authenticate to Cloud Storage to pull down its real startup script (the one that actually sets up the Envoy-based forwarding proxy).
It silently fell back to an unauthenticated download, which only pulled a stub containing basic / scaffolding — noth