Failed `ps:rebuild` leaves old container serving but app image later disappears

Description of problem

Dokku version: 0.38.27

I ran:

dokku ps:rebuild cortland

The image build completed successfully:

#7 writing image sha256:244ee97e20a37323ced78aca407e4f6431be942adc907e74949e365bb631a800 done
#7 naming to docker.io/dokku/cortland:latest
#7 naming to docker.io/dokku/cortland:latest 0.0s done
#7 DONE 1.4s

The new web container then failed its health check:

!     Failure in name='web check': Get "http://172.17.0.3:5000/check.txt": dial tcp 172.17.0.3:5000: connect: connection refused
!     Could not start due to 1 failed checks (web.1)
parallel: This job failed:
/var/lib/dokku/plugins/available/scheduler-docker-local/bin/scheduler-deploy-process-container cortland dockerfile dokku/cortland:latest latest web 1 1
!     builds-record-finalize: no record for cortland/mtw8kwh259qkao, skipping
!     builds-record-finalize: no record for cortland/mtw8kwh259qkao, skipping

!     rebuild command returned 1 errors

Immediately after this, the application was still available. My assumption is that the previously running web container had correctly been left in place after the failed deployment.

Later the application became unavailable. At that point I tried:

dokku ps:restart cortland

and got:

!     App image (dokku/cortland:latest) not found
!      !     App image (dokku/cortland:latest) not found

So the failed rebuild appears to have left the previous container temporarily serving, but without a usable dokku/cortland:latest image from which Dokku could restart the application.

A subsequent dokku ps:rebuild cortland succeeded normally:

Renaming container cortland.web.1.upcoming-17265 (9645245a3edb) to cortland.web.1
-----> Checking for postdeploy task
       No postdeploy task found, skipping
-----> Updated schedule file
-----> Shutting down old containers in 60 seconds
=====> Application deployed:
       http://cortland.historyforge.net
       https://cortland.historyforge.net

!     builds-record-finalize: no record for cortland/mtxi48hahd3c3z, skipping
!     builds-record-finalize: no record for cortland/mtxi48hahd3c3z, skipping

After that successful rebuild:

$ docker images | grep cortland
dokku/cortland:latest   ebcf65b0a2c9   1.59GB   0B   U

$ docker ps -a --filter name=cortland
CONTAINER ID   IMAGE                   COMMAND                  CREATED          STATUS          PORTS   NAMES
9645245a3edb   dokku/cortland:latest   "bundle exec rails s…"   36 minutes ago   Up 36 minutes           cortland.web.1

and:

$ dokku ps:report cortland
=====> cortland ps information
       Deployed:                         true
       Processes:                        1
       Ps can scale:                     true
       Ps computed restart policy:       on-failure:10
       Restore:                          true
       Running:                          true
       Status web 1:                     running (CID: 9645245a3ed)

The host also has the standard Dokku retirement timer enabled:

# /etc/systemd/system/dokku-retire.timer
[Unit]
Description=Run dokku-retire.service every 5 minutes

[Timer]
OnCalendar=*:0/5
Persistent=true

[Install]
WantedBy=timers.target

with:

# /etc/systemd/system/dokku-retire.service
[Unit]
Description=Dokku retire service
Requires=docker.service
After=docker.service

[Service]
Type=oneshot
User=dokku
ExecStart=/usr/bin/dokku ps:retire
SuccessExitStatus=137

The journal shows dokku ps:retire running every five minutes and reporting:

-----> Retiring old containers and images
-----> Retiring expired run containers
-----> Retiring all run containers

Steps to reproduce

I do not have a journal entry proving exactly when the previous Cortland container or image was removed, so I cannot say with certainty that ps:retire caused the outage. However, the observable sequence was:

1. ps:rebuild built a new dokku/cortland:latest image.
2. The candidate container failed its health check.
3. The old container remained available and continued serving the app.
4. Some time later the app became unavailable.
5. dokku ps:restart cortland failed because dokku/cortland:latest did not exist.
6. A fresh ps:rebuild recreated the image and restored the application.

Expected behavior would be that after a failed deployment, the previously working release remains not only running but restartable. It looks as though the failed deployment may leave the old container alive while not restoring/preserving the previous app image/tag, which can turn into an outage once the old container is retired.

I also consistently see this after both failed and successful rebuilds:

builds-record-finalize: no record for cortland/<build-id>, skipping

I do not know whether that is related.

dokku report $APP_NAME

-----> uname: Linux cortland 5.15.0-176-generic #186-Ubuntu SMP Fri Mar 13 11:01:42 UTC 2026 x86_64 x86_64 x86_64 GNU/Linux -----> memory: total used free shared buff/cache available Mem: 957 482 83 24 390 301 Swap: 0 0 0 -----> disk utilization: Filesystem Size Used Avail Use% Mounted on /dev/vda1 25G 14G 11G 57% / /dev/vda15 105M 6.1M 99M 6% /boot/efi -----> disk inode utilization: Filesystem Inodes IUsed IFree IUse% Mounted on /dev/vda1 3.1M 341K 2.8M 11% / /dev/vda15 0 0 0 - /boot/efi -----> docker version: Client: Docker Engine - Community Version: 29.8.0 API version: 1.56 Go version: go1.26.8 Git commit: 88096ef Built: Thu Sep 3 21:50:45 2026 OS/Arch: linux/amd64 Context: default

   Server: Docker Engine - Community
    Engine:
     Version:          29.8.0
     API version:      1.56 (minimum version 1.40)
     Go version:       go1.26.8
     Git commit:       3ce5872
     Built:            Thu Sep  3 21:50:45 2026
     OS/Arch:          linux/amd64
     Experimental:     false
    containerd:
     Version:          v2.3.5
     GitCommit:        1294c24a7da8e5a793ed378161673abe94118892
    runc:
     Version:          1.5.1
     GitCommit:        v1.5.1-0-g8f2685a4
    docker-init:
     Version:          0.19.0
     GitCommit:        de40ad0

-----> docker daemon info: Client: Docker Engine - Community Version: 29.8.0 Context: default Debug Mode: true Plugins: buildx: Docker Buildx (Docker Inc.) Version: v0.37.0 Path: /usr/libexec/docker/cli-plugins/docker-buildx compose: Docker Compose (Docker Inc.) Version: v5.5.1 Path: /usr/libexec/docker/cli-plugins/docker-compose healthcheck: Runs healthchecks against local docker containers (Jose Diaz-Gonzalez) Version: v0.16.0 Path: /usr/libexec/docker/cli-plugins/docker-healthcheck

   Server:
    Containers: 1
     Running: 1
     Paused: 0
     Stopped: 0
    Images: 8
    Server Version: 29.8.0
    Storage Driver: overlay2
     Backing Filesystem: extfs
     Supports d_type: true
     Using metacopy: false
     Native Overlay Diff: true
     userxattr: false
    Logging Driver: json-file
    Cgroup Driver: systemd
    Cgroup Version: 2
    Plugins:
     Volume: local
     Network: bridge host ipvlan macvlan null overlay
     Log: awslogs fluentd gcplogs gelf journald json-file local splunk syslog
    CDI spec directories:
     /etc/cdi
     /var/run/cdi
    Swarm: inactive
    Runtimes: io.containerd.runc.v2 runc
    Default Runtime: runc
    Init Binary: docker-init
    containerd version: 1294c24a7da8e5a793ed378161673abe94118892
    runc version: v1.5.1-0-g8f2685a4
    init version: de40ad0
    Security Options:
     apparmor
      Profile: default
     seccomp
      Profile: builtin
     cgroupns
    Kernel Version: 5.15.0-176-generic
    Operating System: Ubuntu 22.04.5 LTS
    OSType: linux
    Architecture: x86_64
    CPUs: 1
    Total Memory: 957.3MiB
    Name: cortland
    ID: c7ec00f0-97d2-4b52-a362-dccc8aa908a7
    Docker Root Dir: /var/lib/docker
    Debug Mode: false
     File Descriptors: 31
     Goroutines: 49
     System Time: 2026-09-11T22:54:58.517763606Z
     EventsListeners: 0
    Experimental: false
    Insecure Registries:
     ::1/128
     127.0.0.0/8
    Live Restore Enabled: true
    Firewall Backend: iptables
     EnableUserlandProxy: true
     UserlandProxyPath: /usr/bin/docker-proxy

-----> herokuish version: herokuish: v0.11.17 buildpacks: heroku-buildpack-multi v1.2.0 heroku-buildpack-ruby v362 heroku-buildpack-nodejs v366 heroku-buildpack-clojure v98 heroku-buildpack-python v352 heroku-buildpack-java v81 heroku-buildpack-gradle v49 heroku-buildpack-scala v107 heroku-buildpack-php v295 heroku-buildpack-go v236 heroku-buildpack-nginx v33 buildpack-null v3 -----> dokku version: dokku version 0.38.27 -----> dokku-event-listener version: v0.20.1 -----> dokku-update version: dokku-update v0.10.0 -----> docker-container-healthchecker version: v0.16.0 -----> docker-image-labeler version: v0.10.0 -----> git version: git version 2.34.1 -----> lambda-builder version: v0.9.4
-----> netrc version: v0.11.1 ! pack binary is not available -----> plugn version: plugn: v0.17.1 -----> sigil version: v0.12.1 -----> sshcommand version: sshcommand v0.20.2 -----> dokku plugins: 00_dokku-standard 0.38.27 enabled dokku core standard plugin 20_events 0.38.27 enabled dokku core events logging plugin app-json 0.38.27 enabled dokku core app-json plugin apps 0.38.27 enabled dokku core apps plugin builder 0.38.27 enabled dokku core builder plugin builder-dockerfile 0.38.27 enabled dokku core builder-dockerfile plugin builder-herokuish 0.38.27 enabled dokku core builder-herokuish plugin builder-lambda 0.38.27 enabled dokku core builder-lambda plugin builder-nixpacks 0.38.27 enabled dokku core builder-nixpacks plugin builder-null 0.38.27 enabled dokku core builder-null plugin builder-pack 0.38.27 enabled dokku core builder-pack plugin builder-railpack 0.38.27 enabled dokku core builder-railpack plugin buildpacks 0.38.27 enabled dokku core buildpacks plugin builds 0.34.0 enabled dokku core builds plugin caddy-vhosts 0.38.27 enabled dokku core caddy-vhosts plugin certs 0.38.27 enabled dokku core certificate management plugin checks 0.38.27 enabled dokku core checks plugin common 0.38.27 enabled dokku core common plugin config 0.38.27 enabled dokku core config plugin cron 0.38.27 enabled dokku core cron plugin docker-options 0.38.27 enabled dokku core docker-options plugin domains 0.38.27 enabled dokku core domains plugin enter 0.38.27 enabled dokku core enter plugin git 0.38.27 enabled dokku core git plugin haproxy-vhosts 0.38.27 enabled dokku core haproxy-vhosts plugin letsencrypt 0.20.3 enabled Automated installation of let's encrypt TLS certificates logs 0.38.27 enabled dokku core logs plugin network 0.38.27 enabled dokku core network plugin nginx-vhosts 0.38.27 enabled dokku core nginx-vhosts plugin openresty-vhosts 0.38.27 enabled dokku core openresty-vhosts plugin plugin 0.38.27 enabled dokku core plugin plugin ports 0.38.27 enabled dokku core ports plugin proxy 0.38.27 enabled dokku core proxy plugin ps 0.38.27 enabled dokku core ps plugin registry 0.38.27 enabled dokku core registry plugin repo 0.38.27 enabled dokku core repo plugin resource 0.38.27 enabled dokku core resource plugin run 0.38.27 enabled dokku core run plugin scheduler 0.38.27 enabled dokku core scheduler plugin scheduler-docker-local 0.38.27 enabled dokku core scheduler-docker-local plugin scheduler-k3s 0.38.27 enabled dokku core scheduler-k3s plugin scheduler-null 0.38.27 enabled dokku core scheduler-null plugin shell 0.38.27 enabled dokku core shell plugin ssh-keys 0.38.27 enabled dokku core ssh-keys plugin storage 0.38.27 enabled dokku core storage plugin trace 0.38.27 enabled dokku core trace plugin traefik-vhosts 0.38.27 enabled dokku core traefik-vhosts plugin

Additional information

Application is here: https://github.com/historyforge/historyforge App Dockerfile is:

FROM dfurber/historyforge
RUN alias r='bundle exec rails'
RUN addgroup --quiet --gid "32767" "herokuishuser" && \
    adduser \
        --shell /bin/bash \
        --disabled-password \
        --force-badname \
        --no-create-home \
        --uid "32767" \
        --gid "32767" \
        --gecos '' \
        --quiet \
        --home "/app" \
        "herokuishuser" && \
    chown -R herokuishuser /app

USER herokuishuser

CMD ["/bin/bash"]

My deploy procedure is a bit unusual, because I build it locally from this Dockerfile with production assets and then push to Dockerhub, and then the app's Dockerfile just pulls the image. That's because I have 11 installations to deploy, and using the buildpack route or even the local "build it from Dockerfile" route take FOREVER.

FROM ruby:4.0.6

ENV DEBIAN_FRONTEND noninteractive
ENV RAILS_SERVE_STATIC_FILES true
ENV RAILS_LOG_TO_STDOUT true
ENV LANG en_US.UTF-8
ENV LC_ALL en_US.UTF-8

# Install packages
RUN apt-get update -qq && \
  apt-get install -y \
  automake build-essential patch zlib1g-dev liblzma-dev git \
  libssl-dev libreadline-dev \
  libyaml-dev libcurl4-openssl-dev libffi-dev \
  libpq-dev libvips locales postgresql-client && \
  apt-get autoremove -y && \
  apt-get autoclean && \
  apt-get clean && \
  rm -rf /var/lib/apt/lists/* /tmp/* /var/tmp/*

# Generate locale (ENV vars already configure it)
RUN echo "en_US.UTF-8 UTF-8" > /etc/locale.gen && \
    locale-gen en_US.UTF-8

RUN gem install bundler

WORKDIR /app

ADD Gemfile ./Gemfile
ADD Gemfile.lock ./Gemfile.lock

RUN bundle config set without 'development:test' \
    && bundle install --jobs 20 --retry 5 \
    && rm -rf /usr/local/bundle/cache/*.gem \
    && find /usr/local/bundle/gems/ -name "*.c" -delete \
    && find /usr/local/bundle/gems/ -name "*.o" -delete

COPY . .

Thus my deploy command is: docker pull dfurber/historyforge && dokku ps:rebuild --all

I've been doing this for years, and also have many other deployments of other apps using either buildpacks or Dockerfile.

Output of failing commands after running: dokku trace:off

Output of failing commands after running: dokku trace:on