Baike.dev
Log in
Back to tool/Back to issues
#23879·consul

api-gateway xDS ships incomplete snapshots: clusters without endpoints, EDS never finishes

Author: manojrkrishCreated Sep 2, 2026Updated Sep 2, 2026

Overview of the Issue

A Consul native api-gateway with one HTTP listener and one http-route (~1545 backends) does not deliver a complete xDS snapshot.

Envoy gets ~279 partial CDS updates over ~100 minutes. About 25% of clusters never receive EDS, even though those services have healthy Connect instances. The gateway is LIVE during that window and returns no healthy upstream (503).

The same catalog, same Envoy 1.38.2 FIPS, on ingress-gateway converges in 2.0 seconds (1546/1547 clusters, 1 CDS apply, warming_clusters=0). All 391 api-gateway misses got endpoints on ingress-gateway at t=2.002s.

Expected Behavior

api-gateway should push clusters with their endpoints (or an empty ClusterLoadAssignment) in one snapshot, the way ingress-gateway does for the same services.

Actual Behavior

ingress-gateway api-gateway
Clusters 1547 1545
With endpoints 1546 (99.9%) 1158 (75%)
Never resolved 1 (synthetic placeholder) 391
Time to endpoints 2.0 s ~101 min, then stuck
CDS applies 1 279
CDS interval n/a locks to 15.0 s

api-gateway phases (same Envoy process, control_plane.connected_state=1 throughout):

  1. 0–2918s — CDS ramp: 40 → 1545 clusters; endpoints stay ~0
  2. 2918–6045s — EDS drip: 10 → 1144 clusters with endpoints (~0.36/s)
  3. 6045–8004s — tail: 1144 → 1158 (~0.007/s); 387 still empty

Those never-resolved clusters got no EDS — not even empty. Envoy waited initial_fetch_timeout (15s), then activated with 0 hosts. That is the 15.0s CDS cadence.

Suspected cause

api-gateway treats every http-route backend as a first-class upstream (discovery-chain watch + health watch). Consul does not wait for all watches to finish. As soon as one returns, Consul dirties its xDS snapshot and pushes CDS to Envoy — still missing the rest of the clusters/endpoints.

Catalog / EDS events appear to recompile the full discovery-chain set (recompileDiscoveryChains) rather than patch one cluster. CDS apply is Envoy-main-thread (measured 0.3–1.2s). EDS waits. Next catalog event starts another full CDS rebuild.

But ingress-gateway watches one chain and ships one complete snapshot. Same N backends, no drip.

Reproduction Steps

  1. Consul 2.0.3 service mesh (Nomad/VM), ~1500 Connect HTTP services with passing instances.
  2. Create Kind=api-gateway, one HTTP listener, one http-route with those services as backends.
  3. Start consul connect envoy -gateway api -register.
  4. Scrape Envoy admin /stats: cluster_manager.active_clusters, warming_clusters, cds.update_success, and host counts per cluster.
  5. Repeat with ingress-gateway on the same catalog.

Operating system and Environment details

  • Consul 2.0.3
  • Envoy 1.38.2 FIPS (Consul-managed gateway)
  • Nomad, Linux, Consul service mesh
  • One api-gateway, one HTTP listener, TLS via file-system-certificate
  • consul connect envoy -gateway api -register

Consul info for both Client and Server

Client info
agent:
	check_monitors = 1
	check_ttls = 7
	checks = 144
	services = 103
build:
	prerelease = 
	revision = d0f2be93
	version = 2.0.3
	version_metadata = 
consul:
	acl = enabled
	known_servers = 3
	server = false
runtime:
	arch = amd64
	cpu_count = 16
	goroutines = 10937
	max_procs = 16
	os = linux
	version = go1.26.6-X:boringcrypto
serf_lan:
	coordinate_resets = 0
	encrypted = true
	event_queue = 0
	event_time = 1141
	failed = 5
	health_score = 0
	intent_queue = 0
	left = 0
	member_time = 95942
	members = 190
	query_queue = 0
	query_time = 19
Client agent HCL config
Server info
agent:
	check_monitors = 1
	check_ttls = 1
	checks = 8
	services = 11
build:
	prerelease = 
	revision = d0f2be93
	version = 2.0.3
	version_metadata = 
consul:
	acl = enabled
	bootstrap = false
	known_datacenters = 1
	leader = true
	leader_addr = 10.14.88.44:8300
	server = true
raft:
	applied_index = 526873242
	commit_index = 526873242
	fsm_pending = 0
	last_contact = 0
	last_log_index = 526873242
	last_log_term = 1166
	last_snapshot_index = 526857016
	last_snapshot_term = 1165
	latest_configuration = [{Suffrage:Voter ID:b3250461-3187-7d47-c09d-50b9cdbc3689 Address:10.14.88.80:8300} {Suffrage:Voter ID:b0d0bd8b-bb20-b5a5-6fd3-a14c3429b2f3 Address:10.14.88.44:8300} {Suffrage:Voter ID:72a0085d-dc8e-1903-9e0d-a3057b998285 Address:10.14.88.56:8300}]
	latest_configuration_index = 0
	num_peers = 2
	protocol_version = 3
	protocol_version_max = 3
	protocol_version_min = 0
	snapshot_version_max = 1
	snapshot_version_min = 0
	state = Leader
	term = 1166
runtime:
	arch = amd64
	cpu_count = 16
	goroutines = 56884
	max_procs = 16
	os = linux
	version = go1.26.6-X:boringcrypto
serf_lan:
	coordinate_resets = 0
	encrypted = true
	event_queue = 0
	event_time = 1143
	failed = 0
	health_score = 0
	intent_queue = 0
	left = 0
	member_time = 95950
	members = 185
	query_queue = 0
	query_time = 19
serf_wan:
	coordinate_resets = 0
	encrypted = true
	event_queue = 0
	event_time = 1
	failed = 0
	health_score = 0
	intent_queue = 0
	left = 0
	member_time = 1485
	members = 3
	query_queue = 0
	query_time = 1
Server agent HCL config

Log Fragments

Profiler summaries from Envoy admin /stats.

ingress-gateway (same catalog):

samples                 : 17 over 32s
clusters (peak)         : 1547
clusters with endpoints : 1546 (99.9%)
  time to  50%/90%/99%/100% of peak : 2s
endpoint attach rate    : 48.189 clusters/sec
CDS applies observed    : 2
  CDS interval min/med/max: 0.83s / 0.83s / 0.83s
cluster_added / modified: 0 / 0
warming_clusters max / last: 0 / 0

Timeline:

elapsed  clusters  with_endpoints  healthy_eps  warming  cds_ok
    0.0      1547               3            3        0       1
    2.0      1547            1546         2063        0       2
    4.0      1547            1546         2063        0       2
   ... flat through 32s

api-gateway (2072 samples over 8004s):

clusters (peak)         : 1545
clusters with endpoints : 1158 (75.0%)
  time to  50% of peak : 4432s (74m)
  time to  90% of peak : 5682s (95m)   [measured]
  time to  99% of peak : 6297s (105m)
endpoint attach rate    : 0.145 clusters/sec (mean)
  phase 1  CDS ramp     :     0-> 2918s  with_eps     3->   10  0.0024 /s
  phase 2  EDS drip     :  2918-> 6045s  with_eps    10-> 1144  0.3627 /s
  phase 3  tail         :  6045-> 8004s  with_eps  1144-> 1158  0.0071 /s
CDS applies observed    : 293
  CDS interval min/med/max: 0.86s / 14.48s / 30.63s  (phase 1, n=279)
cluster_added           : 40 -> 1545  (+1505, incremental)
cluster_removed         : 0
cluster_updated         : 0 -> 1151  (= endpoint deliveries; with_eps - updated == 3 always)
warming_clusters max    : 73  (non-zero in 99–100% of phase-1 samples)
control_plane.rate_limit_enforced: 0
control_plane.connected_state    : 1 throughout

alloc_logs_api-gateway.stderr.0.txt api-gateway-xds-profile.json api-gateway-xds-profile.ndjson.summary.txt api-gateway-xds-profile.ndjson.convergence.csv

ingress-gateway-xds-profile.json ingress-gateway-xds-profile.ndjson.summary.txt ingress-gateway-xds-profile.ndjson.convergence.csv

Source: hashicorp/consul

View original on GitHubView discussion on GitHub