Consul ACL token is created as root:root 0640 and is unreadable by non-root Docker task user

Nomad version

Nomad 2.0.4 on servers and clients. Consul 1.22.7

This was first observed after upgrading from Nomad 1.11.3 to 2.0.4. We have not yet reproduced this specific ownership behavior on 1.11.3, so it is not confirmed as a regression.

Operating system and Environment details

• RHEL 8 compatible host
• Kernel: 4.18.0-553.139.1.el8_10.x86_64
• Docker task driver
• Consul 1.22.7
• Consul ACLs enabled
• Nomad workload identities used for Consul authentication

Nomad server configuration

All Nomad servers use the following Consul workload identity configuration:

consul {
  address         = "localhost:8501"
  ssl             = true
  ca_file         = "/opt/consul/tls/ca.crt"
  cert_file       = "/opt/consul/tls/agent.crt"
  key_file        = "/opt/consul/tls/agent.key"
  verify_ssl      = true
  grpc_address    = "localhost:8503"
  grpc_ca_file    = "/opt/consul/tls/ca.crt"

  service_identity {
    aud = ["inf293.consul"]
    ttl = "1h"
  }

  task_identity {
    aud  = ["inf293.consul"]
    ttl  = "1h"
    env  = false
    file = true
  }
}

The Consul JWT auth method and binding rules are working. Nomad successfully exchanges the workload identity for a Consul ACL token.

Expected Result

The Consul ACL token written to secrets/consul_token should be readable by the configured task user. Since the task explicitly runs as nobody, we expected either: nobody:<group> 0600 consul_token or another ownership/mode combination that permits UID 65534 to read the token without granting additional groups or running the task as root. Nomad documents secrets/consul_token as being made available to the task. A non-root task should therefore be able to consume it through an application's credentials_file option.

Actual Result

Nomad creates the files as follows:

-rw-r--r--  1 root   root   95 vault_token
-rw-------  1 nobody root  947 nomad_vault_default.jwt
-rw-------  1 nobody root  920 nomad_consul_default.jwt
-rw-r-----  1 root   root   36 consul_token
srw-------  1 nobody root    0 api.sock

The task runs as: uid=65534(nobody) gid=65534(nobody) groups=65534(nobody)

The workload identity JWT files correctly use the configured task user as their owner, but the exchanged Consul ACL token does not. Applications that consume the generated token through a credentials file fail with:

unable to read authorization credentials:
unable to read file /secrets/consul_token:
open /secrets/consul_token: permission denied

Current workaround

For Docker tasks we can add GID 0 as a supplementary group:

config {
  image     = "prom/prometheus:v2.55.1"
  group_add = ["0"]
}

The process remains non-root: uid=65534(nobody) gid=65534(nobody) groups=0(root),65534(nobody) and can then read root:root 0640 files.

However, this broadens the task's access to every root-group-readable bind mount and secret. We do not consider adding the root group to arbitrary non-root workloads an appropriate permanent solution. Running the application as root would be an even less desirable workaround.

Job file (if appropriate)

A minimal reproduction should be possible with:

job "consul-token-permissions" {
  datacenters = ["dc1"]
  type        = "service"

  group "test" {
    task "test" {
      driver = "docker"
      user   = "nobody"

      consul {}

      config {
        image   = "busybox:1.36"
        command = "/bin/sh"
        args = [
          "-ec",
          <<-EOF
            id
            ls -ln /secrets
            stat -c '%U %G %a %n' /secrets/consul_token
            test -r /secrets/consul_token
            sleep 3600
          EOF
        ]
      }
    }
  }
}

thanks, mac