repeated subscribe call failed errors after ACL token deleted
Nomad version
Nomad v1.10.3 BuildDate 2025-07-08T18:46:27Z Revision fde0bf26a7db35fc63ceb50b5b694478034392c6
Operating system and Environment details
Ubuntu 24.04.2 LTS on amd64
Issue
After my allocation has ended, I keep getting subscribe call failed errors in my Consul agent log (the one associated with the Nomad client), which keeeeeps retrying. After the allocation has ended, it should not keep trying to subscribe to the health of services.
(Because of course the ACL token won't be found, Nomad would have invalidated the Consul token when the allocation has stopped.)
Reproduction steps
To be honest, completely reproducing might be tricky since it requires the setup of the Workload Identity, as well as "something" that might cause those ACL problems. Something keeps retrying the service health query despite having exited...
- Run Consul
- Run Nomad with Consul support using the Workload Identity configured
- Have a job file that includes a
templateblock with a{{ range service "some-service" }}snippet - Run the Nomad job
- Stop the allocation
Expected Result
No errors to be found in Consul after the allocation has ended.
Actual Result
Errors like this:
{
"@level": "error",
"@message": "subscribe call failed",
"@module": "agent.rpcclient.health",
"@timestamp": "2025-07-28T09:59:31.614980Z",
"err": "rpc error: code = Unknown desc = ACL not found",
"error": "rpc error: code = Unknown desc = ACL not found",
"failure_count": 17,
"key": "minio-api",
"topic": 1
}Job file (if appropriate)
Nomad Server logs (if appropriate)
Nomad Client logs (if appropriate)
A bunch of things, but the most 'interesting' lines would be these, as the 'kill' is being processed. The Consul service name is minio-api.
2025-07-28T10:12:33.222Z [DEBUG] agent: (runner) stopping watcher
2025-07-28T10:12:33.222Z [DEBUG] agent: (watcher) stopping all views
2025-07-28T10:12:33.223Z [TRACE] agent: (watcher) stopping catalog.services
2025-07-28T10:12:33.223Z [TRACE] agent: (watcher) stopping health.service(minio-api|passing)
2025-07-28T10:12:33.223Z [TRACE] agent: (view) catalog.services stopping poll (received on view stopCh)
2025-07-28T10:12:33.223Z [TRACE] agent: (view) health.service(minio-api|passing) stopping poll (received on view stopCh)
2025-07-28T10:12:33.223Z [TRACE] client.alloc_runner.task_runner.task_hook.script_checks: tasklet executing: alloc_id=4545c919-3fa1-61df-6fa8-0d992b19de96 task=haproxy
2025-07-28T10:12:33.223Z [INFO] agent: (runner) received finish
2025-07-28T10:12:33.223Z [DEBUG] client.alloc_runner.task_runner: task run loop exiting: alloc_id=4545c919-3fa1-61df-6fa8-0d992b19de96 task=haproxy
2025-07-28T10:12:33.236Z [TRACE] client.alloc_runner.runner_hook: proxy to Consul complete: alloc_id=4545c919-3fa1-61df-6fa8-0d992b19de96 src_local=/opt/nomad/data/alloc/4545c919-3fa1-61df-6fa8-0d992b19de96/alloc/tmp/consul_grpc.sock src_remote=@ bytes=268796
2025-07-28T10:12:33.236Z [TRACE] client.alloc_runner.runner_hook: proxy from Consul complete: alloc_id=4545c919-3fa1-61df-6fa8-0d992b19de96 src_local=/opt/nomad/data/alloc/4545c919-3fa1-61df-6fa8-0d992b19de96/alloc/tmp/consul_grpc.sock src_remote=@ bytes=85987
2025-07-28T10:12:33.280Z [TRACE] agent: (view) catalog.services no new data (contents were the same)What I did NOT see, is a reported stop trace message from that service. I do see it with all of the Vault-related templates. Not sure if that is expected.
Source: hashicorp/consul