#26693·nomad

nomad alloc logs -f not stopping after exit 0 ?

Author: dmclfCreated Sep 4, 2025Updated Sep 9, 2026
Labelstype/bugtheme/clitheme/allocation APIstage/acceptedhcc/jira

Question

clarification: what is expected behavior for nomad alloc -f

  1. should nomad alloc -f $ALLOC close after task terminated ?
  2. or should the 'stream' stay open indefinitely, in which case, perhaps from a 'user' perspective

if 1) please handle this as bug report. if 2) no need to read further ... although when there is, example, exit 1, the streaming stops, so it seems inconsistent, but then I guess I will just work with timeout $NR nomad alloc -f to avoid long runtimes in some cases.

Nomad version

Output from nomad version

Nomad v1.10.4
BuildDate 2025-08-12T20:48:32Z
Revision 62b195aaa535b2159d215eaf89e6f4a455d6f686

Nomad v1.9.7
BuildDate 2025-03-11T09:07:15Z
Revision f8695974efdbaa9f80ace06f0eec5bd4d2501035+CHANGES

Operating system and Environment details

Ubuntu 22.04.5

Issue

nomad alloc logs -f $ALLOC seems to stay 'alive' tailing logs, even if the allocation terminated with exit 0.

Reproduction steps

bash
cat << EOF > nomad-hello-world.hcl
job "helloworld" {
  datacenters = ["*"]
  type = "batch"
  group "example" {
    task "alpine" {
      driver = "docker"
      config {
        image = "alpine"
        args = [
          "echo",
          "hello_world",
        ]
      }
    }
  }
}
EOF

nomad job plan nomad-hello-world.hcl
nomad job run -check-index 0 nomad-hello-world.hcl

export MAX_TRIES=20 TRY=0 ALLOC_ID=""

#loop until alloc done downloading image.
while [[ -z "$ALLOC_ID" && $TRY -lt $MAX_TRIES ]]; do
  ALLOC_ID=$(nomad job status helloworld | awk '/run.*(running|complete).*ago/ {getline; print $1}')
  if [[ -z "$ALLOC_ID" ]]; then
    sleep 2
    ((TRY++))
    echo "Waiting... ($TRY/$MAX_TRIES)"
  fi
done

#show alloc status -short
nomad alloc status -short $ALLOC_ID

#echo hello_world should be quick, timeout with 60 seconds.
timeout 60 nomad alloc logs -f $ALLOC_ID

ALLOC_EXITCODE=$?
if test $ALLOC_EXITCODE -eq 0
then 
    echo "ALLOC_EXITCODE:$ALLOC_EXITCODE = OK -> PASS .. logs -f terminated properly"
else
    nomad alloc status $ALLOC_ID
    echo "ERROR ALLOC_EXITCODE=$ALLOC_EXITCODE timeout reached, logs -f was not closing after task dead/terminated/exit (0)"
fi

Expected Result

to get ALLOC_EXITCODE:0 = OK -> PASS .. logs -f terminated properly

Actual Result

ERROR ALLOC_EXITCODE=124 timeout reached, logs -f was not closing after task dead/terminated/exit (0)

+ Job: "helloworld"
+ Task Group: "example" (1 create)
  + Task: "alpine" (forces create)

Scheduler dry-run:
- All tasks successfully allocated.

Job Modify Index: 0
To submit the job with version verification run:

nomad job run -check-index 0 nomad-hello-world.hcl

When running the job with the check-index flag, the job will only be run if the
job modify index given matches the server-side version. If the index has
changed, another user has modified the job and the plan's results are
potentially invalid.


==> View this job in the Web UI: https://nomad-stagingmyinternal-domain.com/ui/jobs/helloworld@default

==> 2025-09-04T09:25:59+08:00: Monitoring evaluation "97ec7bc6"
    2025-09-04T09:25:59+08:00: Evaluation triggered by job "helloworld"
    2025-09-04T09:26:00+08:00: Allocation "5c78ab54" created: node "03e2e1fc", group "example"
    2025-09-04T09:26:00+08:00: Evaluation status changed: "pending" -> "complete"
==> 2025-09-04T09:26:00+08:00: Evaluation "97ec7bc6" finished with status "complete"


==> View job details and metrics in the Web UI: https://nomad-stagingmyinternal-domain.com/ui/jobs/helloworld@default
Error querying allocation: Unexpected response code: 500 (alloc lookup failed: index error: Invalid UUID: encoding/hex: invalid byte: U+0068 'h')
Waiting... (1/20)


==> View job details and metrics in the Web UI: https://nomad-staging.myinternal-domain.com/ui/jobs/helloworld@default
ID       = 5c78ab54-40d2-8191-3e3e-b5d71c736719
Name     = helloworld.example[0]
Created  = 4s ago
Modified = 2s ago

Tasks
Name    State  Last Event  Time                       Lifecycle
alpine  dead   Terminated  2025-09-04T09:26:00+08:00  main


==> View allocation details in the Web UI: https://nomad-staging.myinternal-domain.com/ui/allocations/5c78ab54-40d2-8191-3e3e-b5d71c736719
hello_world
ID                  = 5c78ab54-40d2-8191-3e3e-b5d71c736719
Eval ID             = 97ec7bc6
Name                = helloworld.example[0]
Node ID             = 03e2e1fc
Node Name           = nomad-staging-node1
Job ID              = helloworld
Job Version         = 0
Client Status       = complete
Client Description  = All tasks have completed
Desired Status      = run
Desired Description = <none>
Created             = 1m4s ago
Modified            = 1m2s ago

Task "alpine" is "dead"
Task Resources:
CPU      Memory   Disk     Addresses
100 MHz  300 MiB  300 MiB  

Task Events:
Started At     = 2025-09-04T01:26:00Z
Finished At    = 2025-09-04T01:26:00Z
Total Restarts = 0
Last Restart   = N/A

Recent Events:
Time                       Type        Description
2025-09-04T09:26:00+08:00  Terminated  Exit Code: 0
2025-09-04T09:26:00+08:00  Started     Task started by client
2025-09-04T09:25:59+08:00  Driver      Downloading image
2025-09-04T09:25:59+08:00  Task Setup  Building Task Directory
2025-09-04T09:25:59+08:00  Received    Task received by client


==> View allocation details in the Web UI: https://nomad-staging.myinternal-domain.com/ui/allocations/5c78ab54-40d2-8191-3e3e-b5d71c736719
ERROR ALLOC_EXITCODE=124 timeout reached, logs -f was not closing after task dead/terminated/exit (0)

Job file (if appropriate)

job "helloworld" {
  datacenters = ["*"]
  type = "batch"
  group "example" {
    task "alpine" {
      driver = "docker"
      config {
        image = "alpine"
        args = [
          "echo",
          "hello_world",
        ]
      }
    }
  }
}