Host firewall: policy-selected node is permanently isolated after reboot — host endpoint policy map ends up deny-all (cold boot, audit-off flip) or allow-all (runtime label add), never the computed policy

Is there an existing issue for this?

Closest match is #25448 ("host firewall drops reply traffic for legitimate connections when policies are first enforced"). This report is a stronger manifestation of what looks like the same enforcement-transition problem: in our setup the drop window is not transient — it is fatal and self-sustaining (node permanently isolated until out-of-band reboot), it reproduces on every cold boot, and it also reproduces on a fully synced agent when audit mode is switched off at runtime. Filing separately because the evidence points at the host endpoint's BPF policy map never receiving the computed allow entries, with three distinct end states depending on the path taken.

What happened?

Environment: Cilium 1.18.6 (kubespray), host firewall enabled, kube-proxy replacement enabled, VXLAN tunnel, WireGuard node encryption, bpf-datapath-mode: netkit, CRD identity mode, 61 nodes. A CiliumClusterwideNetworkPolicy with nodeSelector on worker nodes provides ingress lock-down (fromEntities cluster/remote-node/health/host, SSH/ICMP via CiliumCIDRGroup refs plus literal CIDRs, kubelet/NodePort from cluster).

1. Cold boot of a selected node → permanent isolation. ~13 s after cilium-agent starts, every packet in and out is dropped as Policy denied by bpf_host.c:865. The node never recovers: the agent needs the apiserver to finish its sync, and its own egress to the apiserver is what is being dropped. Captured from inside a locked node (systemd unit dumping via crictl exec):

• at agent start, cilium-dbg endpoint list is still empty (host endpoint not created yet) while bpf_host is already attached and enforcing deny-by-default with an empty policy map;
• 105 659 Policy denied drops in 20 s, 73 018 of them the node's own egress to the apiserver LB: identity host->unknown: <node-ip>:* -> <apiserver-lb-ip>:6443;
• at that moment the BPF ipcache is fully populated (8 292 entries) and local CIDR identities for the policy's allowed CIDRs are already allocated — e.g. ICMP from an explicitly allowed CIDR is dropped with its allocated identity (identity 16777222->unknown). The policy is computed; its allow entries are just not in the host endpoint policy map.

2. Runtime audit-off flip on a fully synced agent → same isolation. Boot the node with policy-audit-mode: true (per-node CiliumNodeConfig): node comes up healthy, host endpoint reaches state=ready, realized policy has all 8 L4 ingress rules, enforcement=audit-ingress. Then cilium-dbg endpoint config <hostep> PolicyAuditMode=Disabled → "configuration updated successfully" → the node is fully isolated within seconds. The agent log ends abruptly mid incremental UpdatePolicyMaps processing with no error line — consistent with the deny default being installed while the allow entries never land.

3. Warm enforcement via label add → policy map silently allow-all. With the agent fully synced and the node label added at runtime (CCNP starts selecting the node), the endpoint reports enforcement=ingress, but cilium-dbg bpf policy get <hostep> contains only the wildcard pair (Allow Ingress ANY prefix 0 / Allow Egress ANY prefix 0) — i.e. no actual enforcement. This is the only transition that leaves the node reachable, and it appears to do so by not programming the policy at all. For contrast, a long-running node whose agent processed the CCNP incrementally while already enforcing has a fully programmed map (per-identity allows, port ranges) — so the policy→map pipeline itself works.

So the same policy, same version, produces three different host endpoint policy-map end states depending on the transition path: deny-all (cold boot, audit-off flip), allow-all (runtime label add), correct (incremental update on an already-enforcing endpoint).

Datapath mode does not appear to matter: a cold boot with bpf-datapath-mode: veth (per-node CiliumNodeConfig) locked out identically (caveat: the override's application could not be positively confirmed from inside the locked node).

Impact: any reboot of a policy-selected node isolates it permanently; recovery requires out-of-band console/rescue access. 54 nodes affected in our cluster.

Cilium Version

1.18.6

Kernel Version

6.8.0-124-generic (Ubuntu 24.04)

Kubernetes Version

v1.33.7 (kubespray)

Sysdump

Not attached (production cluster); can provide targeted captures on request. Full monitor drop capture (17 MB) and repeated boot-time snapshots of endpoint list / bpf policy get / bpf ipcache list from inside a locked node are preserved.

Relevant log output

xx drop (Policy denied) flow 0x... to endpoint 0, ifindex 7, file bpf_host.c:865, , identity host->unknown: <node-ip>:39186 -> <apiserver-lb-ip>:6443 tcp SYN
xx drop (Policy denied) flow 0x... to endpoint 0, ifindex 2, file bpf_host.c:865, , identity 16777222->unknown: <allowed-admin-ip> -> <node-ip> icmp EchoRequest

Agent start with audit mode (the boot that survives):

level=warn msg="PolicyAuditMode is enabled. Network policy will not be enforced."
level=info msg="Regenerating restored endpoints" restored=0
level=info msg="Creating host endpoint"

Anything else?

• The reboot lockout window matches #25448's "first enforcement" semantics; unlike #25448 the state is permanent because the dropped traffic includes the agent's own apiserver path (chicken-and-egg).
• Feature combination overlaps with #31209 (tunnel + KPR + WireGuard + host firewall).
• Nothing relevant found in 1.18.7–1.18.13 changelogs.
• Happy to run instrumented builds / additional captures on a dedicated test node.

Code of Conduct

• I agree to follow this project's Code of Conduct