Consul Connect/xDS on client node enters persistent ACL not found state while workload proxy token remains valid

Linked from https://github.com/hashicorp/nomad/issues/28209 Initial inspection suggests this is a Consul bug and requires investigation from the team here.

Nomad v1.11.3 BuildDate 2026-03-11T09:10:27Z Revision 173ab08a0210789da531847c4ce3c3518f7fb34b

Consul v1.22.2 Revision 71f21bf3 Build Date 2025-12-17T05:40:06Z Protocol 2 spoken by default, understands 2 to 3 (agent will automatically use protocol >2 when speaking to compatible agents)

Operating system and Environment details

• OS: RHEL 8.10
• Nomad workloads with Connect sidecars
• Consul workload identities via auth method nomad-workloads
• affected node in the example below: q10i22
• renewal of Consul + Nomad Certs and Tokens via vault-agent

Issue

We are seeing a Consul Connect/xDS failure mode on a Nomad client node where the local Consul client starts returning ACL not found for many Connect-related RPCs such as:

• ConfigEntry.Get
• ConfigEntry.List
• ConfigEntry.ResolveServiceConfig
• Intention.Match
• DiscoveryChain.Get
• ConnectCA.Sign

Once the node enters this state, affected sidecar proxies on that node repeatedly log:

• DeltaAggregatedResources gRPC config stream to local_agent closed: 10, xDS stream terminated due to an irrecoverable error

The important part is: the affected workload proxy token can still be valid and readable via consul acl token read -self.

So this does not currently look like a simple case of an expired or deleted workload token.

Example affected workload

• allocation: 12f978f7-bd96-f83d-339d-8c8fa2f6a97f
• task group: worker-long
• proxy task: connect-proxy-sdsapi-worker
• proxy token accessor: ea36de02-148c-efe5-43c0-cfee1312ad29

The corresponding token is still valid at collection time:

• Description: token created via login: {"requested_by":"nomad_service_sdsapi-worker"}
• AuthMethod: nomad-workloads
• ExpirationTime: null

What we observe

Node-level Consul failure

On q10i22, the local Consul client already shows persistent Connect/ACL failures at the beginning of the collection window:

• 2026-07-02 19:00:01 CEST -> ConfigEntry.Get -> ACL not found
• 2026-07-02 19:00:05 CEST -> ConfigEntry.List -> ACL not found
• 2026-07-02 19:00:10 CEST -> Intention.Match -> ACL not found
• 2026-07-02 19:00:17 CEST -> ConnectCA.Sign -> ACL not found
• 2026-07-02 19:00:17 CEST -> agent.leaf-certs -> ACL not found

Two-phase behavior

We see two phases:

1. Low-rate background phase

   • from about 19:00 to 23:32
   • roughly 1730-1748 ACL not found lines per hour
   • usually around 16-44 per minute

2. Escalation phase

   • starts around 2026-07-02 23:33 CEST
   • error rate jumps sharply
   • hourly counts then rise to:
     • 2026-07-02 23: 27316
     • 2026-07-03 00: 59285

So the node appears to be already unhealthy before the visible error storm begins (but we see no affect on our applications).

Proxy-side effect

The affected proxy then repeatedly logs:

• xDS stream terminated due to an irrecoverable error
• interleaved with:
  • cds: response indicates 0 added/updated cluster(s), 0 removed cluster(s)

This suggests:

• the proxy is still running
• it can still talk to the local agent socket/path
• but the local agent cannot provide stable usable xDS results

We are not claiming that every Connect proxy on the node shows visible xDS errors at exactly the same time. We do have an example on the same node (0ba62444-7268-d158-0304-230b81f3ae62, connect-proxy-vmetrics-pfit) that did not show visible xDS errors at the same time, while the node-level Consul error state was still present.

So the strongest claim we can currently support is:

• the Consul client node enters a broken Connect/ACL state
• some workloads on that node are then visibly affected
• the blast radius may be partial or time-skewed

Why this seems like a bug

The affected workload token is still valid and readable, but the local Consul client continues to fail host-locally with ACL not found across many Connect-related RPCs.

This looks more like:

• broken or stale ACL/cache state in the local client
• or a bad interaction in Connect/xDS/ACL handling on the client node

than a straightforward “the workload token expired”.

Workaround / operational impact

• restarting only the app container is not sufficient
• restarting only the proxy container is not sufficient
• restarting the full allocation restores connectivity temporarily
• the broken node state persist for hours until manual intervention

Reproduction steps

Unfortunately ... starting a Nomad Application with consul connect proxy connection and wait :-/

thanks, Mac