#28467·nomad

Allocation GC fails silently

Author: hashworksCreated Aug 31, 2026Updated Sep 15, 2026

Labelstype/bugstage/waiting-replytheme/client

Nomad version

Nomad v2.0.0
BuildDate 2026-04-21T15:42:13Z
Revision 7a90601f4d9bc473718311fe5167c0c1f66500ba

Operating system and Environment details

Debian

Issue

In some occasions allocations are marked for GC and collected according to the log, but old data remains on disk:

journalctl -u nomad | grep ea308738-40b6-f5c4-9f12-e89d46a9aa1c

2026-07-29T10:01:16.831+0200 [INFO]  client.gc: marking allocation for GC: alloc_id=ea308738-40b6-f5c4-9f12-e89d46a9aa1c
2026-07-30T10:02:45.415+0200 [INFO]  client.gc: garbage collecting allocation: alloc_id=ea308738-40b6-f5c4-9f12-e89d46a9aa1c reason="forced collection"

tree /opt/nomad/data/alloc/ea308738-40b6-f5c4-9f12-e89d46a9aa1c

/opt/nomad/data/alloc/ea308738-40b6-f5c4-9f12-e89d46a9aa1c
├── alloc
│   ├── data
│   ├── logs
│   │   ├── foobar.stderr.0
│   │   └── foobar.stdout.0
│   └── tmp
└── foobar
    ├── local
    ├── private
    │   └── vault_token
    ├── secrets
    │   ├── foo_access_token
    │   ├── bar.jwt
    │   └── vault_token
    └── tmp

mount | grep ea308738-40b6-f5c4-9f12-e89d46a9aa1c

tmpfs on /opt/nomad/data/alloc/ea308738-40b6-f5c4-9f12-e89d46a9aa1c/foobar/secrets type tmpfs (rw,noexec,relatime,size=1024k,inode64)
tmpfs on /opt/nomad/data/alloc/ea308738-40b6-f5c4-9f12-e89d46a9aa1c/foobar/private type tmpfs (rw,noexec,relatime,size=1024k,inode64)

du -hs /opt/nomad/data/alloc/ea308738-40b6-f5c4-9f12-e89d46a9aa1c

56K	/opt/nomad/data/alloc/ea308738-40b6-f5c4-9f12-e89d46a9aa1c

Those allocs result in a 404 if requested by API, but remain in the client state database: bbolt keys /opt/nomad/data/client/state.db allocations ea308738-40b6-f5c4-9f12-e89d46a9aa1c

acknowledged_state
alloc
alloc_consul_acl_token_identities
alloc_identities
task-foobar

This isn't limited to docker allocs but happens with the exec driver as well.

Might be similar to #20544.

Reproduction steps

I'm not sure how to reproduce this yet. In other occasions the GC works just fine.

Temporary mitigation

For now, we're periodically running a script that forcibly removes stale allocs if the disk or inode usage is over 80 percent. Requires a NOMAD_TOKEN with read-job capability on all namespaces.

bash

#!/usr/bin/env bash

NOMAD_ADDR="http://127.0.0.1:4646"
ALLOC_DIR="/opt/nomad/data/alloc"

: "${NOMAD_TOKEN:?NOMAD_TOKEN env var required}"

requires_forced_gc=$(
  df -H / | awk 'NR>1{gsub(/%/,"",$5); if($5+0>80) print "DISK "$5"% "$6}'
  df -i / | awk 'NR>1{gsub(/%/,"",$5); if($5+0>80) print "INODE "$5"% "$6}'
)

[ -z "$requires_forced_gc" ] && exit 0

echo "Disk or inode usage >80%. Forcibly removing stale allocs."

for path in "$ALLOC_DIR"/*/; do
  id=$(basename "$path")
  [ "$id" = "*" ] && continue

  code=$(curl -s -o /dev/null -w '%{http_code}' \
    -H "X-Nomad-Token: $NOMAD_TOKEN" \
    "$NOMAD_ADDR/v1/allocation/$id")

  if [ "$code" = "404" ]; then
    echo "Removing stale alloc $id"
    grep "/opt/nomad/data/alloc/${id:?}" /proc/mounts | awk '{print $2}' \
      | sort -r | xargs -r -n1 umount -l
    if [ "$?" == 1 ]; then
	echo "Failed to lazily detach the remaining alloc filesystems";
    else
         rm -Rf "${ALLOC_DIR:?}/${id:?}"
        if [ "$?" == 1 ]; then
	    echo "Failed to remove the alloc data dir";
        fi
    fi
  else
    echo "Keeping $id (HTTP status code $code)"
  fi
done

Source: hashicorp/nomad

View original on GitHub View discussion on GitHub