#28578·nomad

docker driver: `total_ticks_count` under-counts CPU since 1.9.5 (stats poll period is ~2.5 s, counter incremented as if once per second)

Author: andriimredoclyCreated Sep 18, 2026Updated Sep 18, 2026

Nomad version

2.0.4 (also reproduced on 1.9.5 and 1.11.3; correct on 1.7.7, 1.8.4, 1.9.3).

Operating system and Environment details

Docker driver, Docker 23.0.2 and 29.1.3, cgroup v2, Ubuntu 24.04 arm64 (AWS Graviton, 2500 MHz per core). Client telemetry:

hcl

telemetry {
  collection_interval        = "1s"
  publish_allocation_metrics = true
  prometheus_metrics         = true
}

Issue

nomad.client.allocs.cpu.total_ticks_count was added in #17579 so that rate() over it gives a task's average MHz regardless of scrape timing. Since 1.9.5 it reports 0.4 to 0.5 of the real consumption.

Two things combine:

client/allocrunner/taskrunner/task_runner.go (setGaugeForCPU) calls IncrCounterWithLabels(total_ticks_count, TotalTicks) on every stats update. TotalTicks is a rate (MHz over Docker's ~1 s sample window), not an amount, and no elapsed time is applied. The counter is therefore "MHz × number of updates", which equals MHz × seconds only when updates arrive exactly once per second.
#24768 (1.9.5) switched drivers/docker/stats.go to a non-streaming ContainerStats call so that PreCPUStats is populated. That call blocks until Docker has taken two samples (~1 to 1.5 s), and collectStats re-arms the ticker only after the call returns. The effective update period is collection_interval + call latency: ~2.0 s with Docker 29 and ~2.5 s with Docker 23. Both the gauge and the counter now update at that cadence. The gauge still averages correctly because it holds its last value; the counter loses the unaccounted seconds.

Reproduction steps

Run a docker task that pins one core (job file below) on a node with collection_interval = "1s".
Read the client's /v1/metrics?format=prometheus every second and record nomad_client_allocs_cpu_total_ticks and nomad_client_allocs_cpu_total_ticks_count for the task.
Read the task's cgroup cpu.stat usage_usec at the start and at the end as ground truth.

Expected Result

Gauge ≈ 2500; counter grows by ≈ 2500 per second; rate(nomad_client_allocs_cpu_total_ticks_count[1m]) ≈ avg_over_time(nomad_client_allocs_cpu_total_ticks[1m]) ≈ cgroup usage × 2500.

Actual Result

Nomad	update gap	gauge mean	counter rate	counter ÷ gauge	cgroup
1.7.7, 1.8.4, 1.9.3 (Docker 29.1.3)	0.99 s	2506	2522	1.01	1.000 core
1.9.5, 1.11.3, 2.0.4 (Docker 29.1.3)	2.00 s	2507	1253	0.50	1.000 core
2.0.4 (Docker 23.0.2)	2.49 s	2497	1007	0.40	0.997 core

Per-second samples show the counter moving by the full gauge value (2464 to 2541) on 40 to 50 % of seconds and by 0 on the rest, e.g. 0, 2486, 0, 0, 2504, 0, 2507, ….

Under a periodic load (1 s busy, 4 s idle, five identical allocations) the counter reads a consistent 0.40 of the true 500 MHz on every allocation, while the gauge scraped at 5 s aliases to anywhere between 0.5× and 2× per allocation.

1.9.0 and 1.9.4 are omitted because they report a wrong CPU value in both metrics (#24224, fixed by #24768). This report is about the update cadence and the counter arithmetic that #24768 exposed.

Reproduction with Docker Compose

Self-contained, one file, runs on Docker Desktop (needs Compose >= 2.23 for inline configs). Two nomad agent -dev containers, 1.8.4 and 2.0.4, share the host Docker daemon and run the same job; Prometheus scrapes both every second and prefixes every metric name with the version so both can be graphed in one query.

docker-compose.yml

yaml

# Reproduction for hashicorp/nomad#28578: two `nomad agent -dev` containers
# (1.8.4 = correct, 2.0.4 = broken) on one Docker daemon, same burn job,
# Prometheus scraping both every second with the version prefixed to every
# metric name. Needs Docker Compose >= 2.23 (inline `configs`).
#
#   docker compose up -d
#   docker compose exec nomad-good nomad job run /jobs/burn.hcl
#   docker compose exec nomad-bad  nomad job run /jobs/burn.hcl
#   # wait two minutes, then open http://localhost:9091 and run the queries below
#   docker compose down -v; docker ps -aq --filter name=burn- | xargs docker rm -f

x-nomad: &nomad
  privileged: true
  volumes:
    - /var/run/docker.sock:/var/run/docker.sock:rw
    - /sys/fs/cgroup:/sys/fs/cgroup:rw
  configs:
    - source: nomad_config
      target: /nomad/config.hcl
    - source: burn_job
      target: /jobs/burn.hcl

services:
  nomad-good:
    <<: *nomad
    image: hashicorp/nomad:1.8.4
    command: agent -dev -config=/nomad/config.hcl -bind=0.0.0.0 -data-dir=/tmp/nomad-good
    ports: ["14646:4646"]
  nomad-bad:
    <<: *nomad
    image: hashicorp/nomad:2.0.4
    command: agent -dev -config=/nomad/config.hcl -bind=0.0.0.0 -data-dir=/tmp/nomad-bad
    ports: ["24646:4646"]
  prometheus:
    image: prom/prometheus:v3.5.0
    ports: ["9091:9090"]
    configs:
      - source: prometheus_config
        target: /etc/prometheus/prometheus.yml

configs:
  nomad_config:
    content: |
      datacenter = "dc1"
      acl { enabled = false }
      telemetry {
        collection_interval        = "1s"
        disable_hostname           = true
        prometheus_metrics         = true
        publish_allocation_metrics = true
      }
      client {
        enabled = true
        # 2500 MHz x vCPUs of the Docker VM, so one pinned core reads 2500.
        # Docker Desktop's VM has no CPU clock Nomad can fingerprint. Adjust to your core count.
        cpu_total_compute = 17500
      }
  burn_job:
    content: |
      job "cpu-probe" {
        datacenters = ["dc1"]
        group "probe" {
          task "burn" {
            driver = "docker"
            config {
              image   = "node:22-alpine"
              command = "node"
              args    = ["-e", "for(;;){}"]
            }
            resources {
              cpu    = 3000
              memory = 128
            }
          }
        }
      }
  prometheus_config:
    content: |
      global: { scrape_interval: 1s }
      scrape_configs:
        - job_name: nomad_good
          metrics_path: /v1/metrics
          params: { format: [prometheus] }
          static_configs: [{ targets: [nomad-good:4646], labels: { nomad_version: "1.8.4" } }]
          metric_relabel_configs:
            - { source_labels: [__name__], regex: "nomad_(.*)", target_label: __name__, replacement: "v1_8_4_nomad_$${1}" }
        - job_name: nomad_bad
          metrics_path: /v1/metrics
          params: { format: [prometheus] }
          static_configs: [{ targets: [nomad-bad:4646], labels: { nomad_version: "2.0.4" } }]
          metric_relabel_configs:
            - { source_labels: [__name__], regex: "nomad_(.*)", target_label: __name__, replacement: "v2_0_4_nomad_$${1}" }

bash

docker compose up -d
docker compose exec nomad-good nomad job run /jobs/burn.hcl
docker compose exec nomad-bad  nomad job run /jobs/burn.hcl

After two minutes, in Prometheus at http://localhost:9091:

promql

rate(v1_8_4_nomad_client_allocs_cpu_total_ticks_count{task="burn"}[1m]) or rate(v2_0_4_nomad_client_allocs_cpu_total_ticks_count{task="burn"}[1m])
avg_over_time(v1_8_4_nomad_client_allocs_cpu_total_ticks{task="burn"}[1m]) or avg_over_time(v2_0_4_nomad_client_allocs_cpu_total_ticks{task="burn"}[1m])
changes(v1_8_4_nomad_client_allocs_cpu_total_ticks_count{task="burn"}[1m]) or changes(v2_0_4_nomad_client_allocs_cpu_total_ticks_count{task="burn"}[1m])

Result on Docker Desktop 29.1.3, 7 vCPU, cgroup v2 (cpu_total_compute = 17500, so one pinned core = 2500):

	1.8.4	2.0.4
`rate(total_ticks_count[1m])`	2496	930
`avg_over_time(total_ticks[1m])`	2496	2494
counter updates per minute	59	22

Cleanup: docker compose down -v; docker ps -aq --filter name=burn- | xargs docker rm -f.

Job file (if appropriate)

hcl

job "cpu-probe" {
  datacenters = ["dc1"]

  group "probe" {
    task "burn" {
      driver = "docker"

      config {
        image   = "node:22-alpine"
        command = "node"
        args    = ["-e", "for(;;){}"]
      }

      resources {
        cpu    = 3000
        memory = 128
      }
    }
  }
}

Nomad Server logs (if appropriate)

Not relevant; nothing is logged.

Nomad Client logs (if appropriate)

No stats errors are logged during collection (error collecting stats from container count: 0).

Source: hashicorp/nomad

View original on GitHub View discussion on GitHub