L7 regex NACK loop starves the agent API and blocks all pod creation on the node

Is there an existing issue for this?

• I have searched the existing issues

Version

equal or higher than v1.20.1 and lower than v1.21.0

What happened?

A single L7 CiliumNetworkPolicy made the Cilium agent unusable on two worker nodes.

One path regex in the policy compiles to an RE2 program of 134 instructions. Envoy's default cap is 100, so Envoy NACKs the config. It rejects the whole cilium.NetworkPolicy resource, not just the offending rule.

The agent then re-sends the same config immediately. Envoy NACKs again. The loop runs at roughly 500 xDS exchanges per second and never backs off.

Consequences on the affected node:

• agent CPU goes to 282m, against 52-77m on healthy nodes in the same cluster ; hosts load average increases strongly
• the agent stops regenerating endpoints. 18 of 195 endpoints went not-ready, including pods with no network policy at all
• no new pod can start on the node. The CNI plugin times out against the agent API: failed to create pod network sandbox ... plugin type="cilium-cni" failed (add): unable to create endpoint: Cilium API client timeout exceeded

The blast radius is the node, not the namespace. Workloads unrelated to the policy went down: argocd-server, argocd-application-controller, elasticsearch, trivy-operator, reloader, snapshot-controller. Three unrelated production services were degraded.

Deleting the CiliumNetworkPolicy fixed it instantly. Restarting the agent does not: the rejected config is pushed again as soon as it comes back.

Expected behaviour: a config Envoy rejects should be dropped, or retried with backoff, and it should never degrade endpoints that have nothing to do with it. Ideally the policy is rejected at admission time so it never reaches the agent.

#41314 reports the same trigger, but describes the impact as "all policies for the endpoint are dropped" (fail-open, a security problem). This is a different failure mode: the node stops accepting pods. One comment there (niklasfrick, 2025-08-22) describes an agent OOMKill that "made the whole node unresponsive" and concludes "Not sure if this behavior is directly related" - it is.

#41314 was closed by #46820, which adds a Helm option to raise the Envoy limit. That moves the threshold, it does not remove the loop. A slightly larger regex reproduces the same outage.

How can we reproduce the issue?

1. Create a target pod:

apiVersion: v1
kind: Pod
metadata:
  name: target
  labels:
    app: target
spec:
  containers:
    - name: nginx
      image: nginx

1. Apply this policy. The regex is taken verbatim from the policy that caused our outage - an allowlist of URL paths for a DefectDojo frontend. Nothing exotic, just an alternation of 11 path segments:

apiVersion: cilium.io/v2
kind: CiliumNetworkPolicy
metadata:
  name: l7-oversized-regex
spec:
  endpointSelector:
    matchLabels:
      app: target
  ingress:
    - fromEntities:
        - cluster
      toPorts:
        - ports:
            - port: "80"
              protocol: TCP
          rules:
            http:
              - method: GET
                path: "/(dashboard|alerts|calendar|components|metrics|critical_asset_metrics|critical_product_metrics|simple_search|support|celery_status|choices)([/?].*)?"

The endpointSelector must match a real pod. If it matches nothing, no Envoy config is generated and the bug does not trigger.

1. Watch Envoy reject it in a loop:

kubectl logs -n kube-system -l k8s-app=cilium-envoy --since=1m | grep -c rejected

1. Compare agent CPU on that node against the others:

kubectl top pod -n kube-system -l k8s-app=cilium

1. Count xDS churn - it is local to the affected node:

kubectl logs -n kube-system <cilium-agent-on-that-node> --since=5s | grep -c OnStreamRequest

1. Schedule any new pod on that node. It stays in ContainerCreating with FailedCreatePodSandBox.

Step 6 is the point: the pod in step 6 has nothing to do with the policy in step 2.

Cilium Version

Client: 1.20.1 7d68cfb3 2026-08-18T07:45:23Z go version go1.26.5 linux/amd64

Daemon: 1.20.1 7d68cfb3 2026-08-18T07:45:23Z go version go1.26.5 linux/amd64

Images:

• quay.io/cilium/cilium:v1.20.1
• quay.io/cilium/cilium-envoy:v1.37.5-1786810558-766ccfb37260a43e9d228837aa84ce3faf9f64e7

Relevant config: external-envoy-proxy: true, enable-envoy-config: true, kube-proxy-replacement: true, routing-mode: tunnel (vxlan), enable-gateway-api: true, enable-host-firewall: true, enable-wireguard: true, identity-allocation-mode: crd.

No re2.max_program_size override is set - cilium-envoy-config and cilium-config are at their defaults. This is a stock 1.20.1 install.

Kernel Version

7.0.0-31-generic, Ubuntu 26.04.1 LTS Cloud Minimal

Kubernetes Version

v1.36.4

Regression

Not verified on our side. We have only run this on 1.20.1. The comment on #41314 suggests it appeared between v1.17.3 and v1.17.6.

Sysdump

No response

Relevant log output

Envoy, rejecting the same resource three times in 5 ms:

[2026-09-15 10:44:29.712][2][warning][config] [external/envoy/source/extensions/config_subscription/grpc/grpc_subscription_impl.cc:138] gRPC config for type.googleapis.com/cilium.NetworkPolicy rejected: regex '/(dashboard|alerts|calendar|components|metrics|critical_asset_metrics|critical_product_metrics|simple_search|support|celery_status|choices)([/?].*)?' RE2 program size of 134 > max program size of 100 set for the error level threshold. Increase configured max program size if necessary.
[2026-09-15 10:44:29.714][2][warning][config] [external/envoy/source/extensions/config_subscription/grpc/grpc_subscription_impl.cc:138] gRPC config for type.googleapis.com/cilium.NetworkPolicy rejected: regex '/(dashboard|alerts|calendar|components|metrics|critical_asset_metrics|critical_product_metrics|simple_search|support|celery_status|choices)([/?].*)?' RE2 program size of 134 > max program size of 100 set for the error level threshold. Increase configured max program size if necessary.
[2026-09-15 10:44:29.717][2][warning][config] [external/envoy/source/extensions/config_subscription/grpc/grpc_subscription_impl.cc:138] gRPC config for type.googleapis.com/cilium.NetworkPolicy rejected: regex '/(dashboard|alerts|calendar|components|metrics|critical_asset_metrics|critical_product_metrics|simple_search|support|celery_status|choices)([/?].*)?' RE2 program size of 134 > max program size of 100 set for the error level threshold. Increase configured max program size if necessary.


Agent, on the same node:

level=warn msg="NACK received, reverting resource changes" module=agent.controlplane.envoy-proxy xdsTypeURL=type.googleapis.com/cilium.NetworkPolicy version=6b4f795
level=warn msg="NACK received, reverting resource changes" module=agent.controlplane.envoy-proxy xdsTypeURL=type.googleapis.com/cilium.NetworkPolicy version=7d9995c
level=warn msg="Failed to apply L7 proxy policy changes. These will be re-applied in future updates." module=agent.controlplane.endpoint-manager error="context canceled"
level=warn msg="Creation of endpoint failed" ciliumEndpointName=trivy-system/trivy-operator-764975d447-7vw7v
level=warn msg="Creation of endpoint failed" ciliumEndpointName=reloader/reloader-6b689b9df8-q868k
level=info msg="generating BPF for endpoint failed, keeping stale directory" ciliumEndpointName=argocd/argocd-server-868c9b447c-p7fnb
level=error msg="Endpoint regeneration failed" ciliumEndpointName=argocd/argocd-server-868c9b447c-p7fnb

The agent also floods its own log with the xDS exchange, which rotates out any earlier context - 2472 OnStreamRequest/OnStreamResponse lines in 5 seconds on the affected node, 0 on the others.

Kubelet, for any pod scheduled on that node afterwards:

Warning  FailedCreatePodSandBox  kubelet  Failed to create pod sandbox: rpc error: code = Unknown desc = failed to create pod network sandbox k8s_<pod>_<ns>_<uid>_0: error adding pod <ns>_<pod> to CNI network "cilium": plugin type="cilium-cni" failed (add): unable to create endpoint: Cilium API client timeout exceeded

Anything else?

This is worse than a config error : The policy is valid YAML. The API server accepts it. kubectl apply --dry-run=server accepts it. ArgoCD reports a successful sync. Nothing in a normal GitOps workflow catches it. Any developer with commit access to an application repository can take down the cluster network, with no elevated privileges and no bad intent.

Diagnosis is misleading. Every visible symptom points at a failing node or a broken CNI. The only place the real cause appears is the Envoy log. We spent a long time on the wrong node before finding it.

On validating the regex up front : #41314 concluded that this cannot be validated because Go does not use RE2 and regexp exposes no program size. That is true for an exact figure, but a usable upper bound is not hard. We measured the marginal cost of each construct against libre2:

Length alone is not a bound: /x{1000} is 8 characters and compiles to 1005. \p{...} is worse.

This heuristic holds up:

size(rx) <= 80 && !rx.contains('{') && !rx.contains('\p') && !rx.contains('\P') &&
size(rx) + 7*count(rx, '.') + 9*count(rx, '\') <= 100

Fuzzed over 180215 randomly generated valid regexes, comparing the verdict against the real program size from libre2: zero cases accepted by the heuristic that exceed 100. Zero false positives against 23 real-world path regexes from our own policies.

Stopgap we deployed a ValidatingAdmissionPolicy that rejects the policy at admission time. It is a workaround for the trigger, not a fix for the loop, and it is certainly imperfect - it only covers L7 path, method and host regexes. It does nothing about other ways to overload cilium-envoy or the agent: large DNS rulesets, identity count, broad toFQDNs.

Hopefully useful to someone else hitting this issue :

apiVersion: admissionregistration.k8s.io/v1
kind: ValidatingAdmissionPolicy
metadata:
  name: deny-oversized-l7-regex
spec:
  failurePolicy: Fail
  matchConstraints:
    matchPolicy: Equivalent
    resourceRules:
      - apiGroups: ["cilium.io"]
        apiVersions: ["v2"]
        operations: ["CREATE", "UPDATE"]
        resources: ["ciliumnetworkpolicies", "ciliumclusterwidenetworkpolicies"]
  validations:
    - expression: |
        (
          (has(object.spec) ? [object.spec] : []) +
          (has(object.specs) ? object.specs : [])
        ).all(sp,
          (
            (has(sp.ingress) ? sp.ingress : []) +
            (has(sp.egress) ? sp.egress : [])
          ).all(rule,
            (has(rule.toPorts) ? rule.toPorts : []).all(tp,
              (has(tp.rules) && has(tp.rules.http) ? tp.rules.http : []).all(h,
                [
                  has(h.path) ? h.path : '',
                  has(h.method) ? h.method : '',
                  has(h.host) ? h.host : ''
                ].all(rx,
                  size(rx) <= 80 &&
                  !rx.contains('{') &&
                  !rx.contains('\\p') &&
                  !rx.contains('\\P') &&
                  size(rx)
                    + 7 * (size(rx.split('.')) - 1)
                    + 9 * (size(rx.split('\\')) - 1) <= 100
                )
              )
            )
          )
        )
      message: "L7 regex too complex. Envoy caps each compiled RE2 program at 100; a larger one makes Envoy NACK the whole policy and the agent reapplies it in a loop until its API starves and no pod can start on the node. Keep every http path, method and host at 80 characters or less, without bounded repetition {n,m} and without Unicode classes \\p or \\P. Unescaped dots and backslash classes are charged extra: write [.] rather than . for a literal dot, and split a long alternation into several shorter rules."
---
apiVersion: admissionregistration.k8s.io/v1
kind: ValidatingAdmissionPolicyBinding
metadata:
  name: deny-oversized-l7-regex
spec:
  policyName: deny-oversized-l7-regex
  validationActions:
    - Deny
  matchResources:
    namespaceSelector: {}

Related: #41314, #41027, #46820.

AI disclosure, per the Cilium AI policy : this writeup was drafted with AI assistance from my own incident data. The outage, the reproduction steps, the RE2 measurements, the fuzzing results and the admission policy are mine, taken from a production cluster, and I have reviewed and verified everything above before posting.

Cilium Users Document

• Are you a user of Cilium? Please add yourself to the Users doc

Code of Conduct

• I agree to follow this project's Code of Conduct