Gateway API host-network: changing a listener port wedges Envoy in a permanent NACK loop that survives all restarts

Is there an existing issue for this?

• I have searched the existing issues

Closest existing issues, both of which I think are distinct:

• #45913 — TLS Passthrough listener on a non-standard port silently never binds. Same underlying listener-collapsing behaviour, different symptom (never binds, versus a permanent wedge after a change).
• #34982 — one invalid listener stops updates for the others (closed as not planned, v1.16). Related to a secondary effect I hit, but it does not describe the persistence across restarts or the address-set cause.

What happened?

In Gateway API host-network mode, Cilium collapses every listener of a Gateway into a single Envoy listener whose address set carries all the ports (additionalAddresses). Changing any listener's port therefore changes that collapsed listener's address set, and Envoy cannot update a listener's addresses in place. It rejects the update:

error adding listener: 'kube-system/cilium-gateway-default/listener' has duplicate
address '0.0.0.0:80,[::]:80,0.0.0.0:443,[::]:443' as existing listener

The rejection then becomes permanent. It survives restarting Envoy, restarting the agent, restarting both in either order, and restarting the operator. A fresh Envoy connects, binds the old ports, receives the current config, NACKs it, and retries every 15 seconds indefinitely.

Deleting the CiliumEnvoyConfig does not help either: the operator regenerates it within seconds, so the agent appears to coalesce the delete and recreate into an update of the same resource name, and no removal ever reaches Envoy.

What I think is going on

My reading — and I would not be surprised to be wrong about the mechanism, since I inferred it from behaviour rather than from the source — is that the agent's xDS server replays the last ACKed version to each new Envoy stream before offering the current one. That would explain why a fresh Envoy binds the old ports before NACKing the new config, and why no amount of restarting escapes it: every new stream re-establishes the same stale state that caused the conflict.

What I am confident about is the observable part: the address set changes, Envoy refuses the in-place update, and nothing short of removing the Gateway clears it.

What does clear it

Deleting the Gateway itself (not the CEC). The operator then tears the CEC down for real, the agent propagates an actual removal, and Envoy acks it — lds: remove listener, ports unbind, ack state clears. Re-applying the Gateway afterwards arrives as a clean ADD carrying the full address set, and every port binds first try in about ten seconds.

Routes reattach by parentRef name, so deleting and recreating a same-named Gateway needs no route changes. The gateway serves nothing during the window, which was acceptable in my case only because the cutover already had the edge down.

A related trap while diagnosing this

Gateway.status.listeners reports Programmed=True for the whole duration of the wedge. That condition appears to reflect config generation rather than whether a socket exists, so every status signal looks healthy while nothing is bound. netstat on the node and an actual request were the only things that told the truth. If listener status could reflect the Envoy ACK state, this class of problem would be far easier to spot — though I appreciate that may be a larger change than it sounds.

How to reproduce

1. Cilium with gatewayAPI.enabled=true and gatewayAPI.hostNetwork.enabled=true
2. A Gateway with an HTTP listener on :80 and an HTTPS listener on :443, both bound and serving
3. Change one listener's port (in my case adding a listener that shifted the collapsed set)
4. Observe the agent log — completing NACK — and Envoy — gRPC config ... rejected
5. Restart Envoy, the agent, and the operator in any order; the NACK loop resumes each time
6. Delete the Gateway and re-apply it; everything binds immediately

Cilium Version

Observed on 1.20.0. The cluster has since moved to 1.20.1 and I have not deliberately re-wedged it to confirm the behaviour is unchanged there, since it is now serving real traffic — so treat the 1.20.1 status as untested rather than fixed.

Kernel Version

Talos Linux v1.13.x, single-node cluster, one public IPv4.

Kubernetes Version

v1.36.x

Regression

Not known to be a regression — this was the first time I had used host-network mode, so I have no earlier version to compare against.

Anything else?

Two consequences I have written into my own runbook, in case they are useful for the docs:

• Adding a listener on an existing port is safe, because the collapsed address set does not change — only the filter chains do.
• Adding or changing a port is effectively a maintenance event rather than a config tweak, and should be planned like a restart.

Happy to gather more detail, test a patch, or re-run the reproduction on a scratch cluster if that would help — I have the environment to hand.