v1.19.7: intermittent pod to ClusterIP (kube-apiserver) timeouts on native routing with kube-proxy-replacement=false, node-dependent

Is there an existing issue for this?

• I have searched the existing issues

This looks like it may share a root cause with #48178 (post-#47204 regression with kube-proxy-replacement=false), but that report is explicitly scoped to tunnel/overlay clusters and states the #45252 fix "resolves the regression for native-routing clusters". This report is a native-routing cluster, so it is filed separately as a distinct data point rather than as a comment there.

Version

equal or higher than v1.19.7 and lower than v1.20.0

What happened?

After upgrading a GKE cluster from Cilium 1.18.10 to 1.19.7, pods began experiencing intermittent TCP timeouts to the default/kubernetes ClusterIP. The failure is node-dependent: 2 of 4 nodes were affected, the other 2 were completely clean.

The service in question is the standard kube-apiserver ClusterIP, whose only backend is the GKE control plane endpoint, which lives outside the cluster in the node subnet:

ID   Frontend             Service Type   Backend
76   10.72.96.1:443/TCP   ClusterIP      1 => 10.72.0.46:443/TCP (active)

Note the address layout, which may be relevant given #47204 changed which frontends are programmed when KPR is off. In this cluster the node subnet contains all of: node IPs (10.72.0.12, .18, .64, .72), the control plane endpoint (10.72.0.46), and the VIPs of four internal LoadBalancer services (10.72.0.49, .63, .66, .67, all externalTrafficPolicy: Cluster). Those LoadBalancer frontends do appear in cilium-dbg service list, with ID 0.

Observed failure rates from a fresh pod on each node, 8 attempts each, curl to https://kubernetes.default.svc/version with the ServiceAccount token and cluster CA:

Both affected nodes returned to 100% success immediately after rolling back to 1.18.10, on the same nodes, same pods, same probe. A second post-rollback round of 10 attempts per node was also 40/40. A separate cluster running 1.16.19 with the same values was 72/72 across 9 nodes.

Real workloads were affected, not just the probe. On node D, external-secrets failed its leader-election lease against the apiserver:

Error retrieving lease lock ... Get "https://10.72.96.1:443/apis/coordination.k8s.io/v1/..."

along with reconciler errors in other API-dependent controllers. These errors began ~1 minute after the Cilium agent restarted on that node and stopped when the rollback completed. There were zero such errors in the 90 minutes before the upgrade.

Two things that may help narrow it down:

1. No Cilium drops were recorded at all. cilium-dbg metrics list showed only reason=Unsupported L3 protocol (7 egress / 46 ingress), nothing resembling a service or policy drop. cilium-dbg monitor --type drop was silent while the failing traffic was being generated.
2. Datapath programming was identical between an affected and an unaffected node: both had 137 entries in cilium-dbg service list and 169 in cilium-dbg bpf lb list, with the apiserver frontend correctly mapped on both, and no frontend collision on 10.72.0.46.

So from Cilium's own accounting the packets are not being dropped, yet the connection does not complete, which is consistent with the forward and reply paths being split between the BPF path and the host/kube-proxy path as described in #48178.

How can we reproduce the issue?

Not reliably reduced to a minimal repro yet. Conditions present here:

• GKE with kube-proxy-replacement=false and cni.chainingMode=portmap (kube-proxy still handling services)
• Native routing (ipv4NativeRoutingCIDR covering the pod CIDR only), enable-bpf-masquerade=true
• A ClusterIP whose backend is outside the cluster and inside the node subnet (the GKE control plane endpoint)
• Internal LoadBalancer services with VIPs allocated from that same node subnet

From a fresh pod, repeatedly open new connections to the kube-apiserver ClusterIP. On affected nodes 25-75% of new connections time out. Existing long-lived connections appear unaffected, which is why workload impact lagged the upgrade by hours until pods were rescheduled.

Cilium Version

v1.19.7 (Helm chart 1.19.7). Previously working on v1.18.10.

Kernel Version

6.12.94+ (Container-Optimized OS from Google)

Kubernetes Version

v1.34.9-gke.1610001

Regression

Yes. Worked on 1.18.10 and 1.16.19, broke on 1.19.7. Given #47204 shipped in 1.19.7 and changed which service map entries are programmed when kube-proxy-replacement=false, that is the most likely candidate, as it is in #48178.

Sysdump

Not captured. The cluster was rolled back to restore service before a sysdump or a packet-level trace was taken, so I do not have the cilium monitor -v flow trace or host conntrack dump that #48178 includes. That is the main gap in this report and I want to be upfront about it: the version A/B is strong, but I have no packet-level evidence of the mechanism.

I can re-upgrade a non-production cluster to 1.19.7 to capture cilium monitor -v, host conntrack for the stuck flow, and a full sysdump if that would be useful. Happy to test a patch or a candidate build.

Relevant log output

# Cilium's view of the apiserver service on an affected node (no anomaly visible)
$ cilium-dbg bpf lb list | grep 10.72.96.1
10.72.96.1:443/TCP (0)   0.0.0.0:0 (76) (0) [ClusterIP, non-routable]
10.72.96.1:443/TCP (1)   10.72.0.46:443/TCP (76) (1)

# Drop counters on the affected node while failing traffic was being generated
$ cilium-dbg metrics list | grep drop
cilium_drop_count_total  direction=EGRESS   reason=Unsupported L3 protocol   7
cilium_drop_count_total  direction=INGRESS  reason=Unsupported L3 protocol   46