Baike.dev
Log in
Back to tool/Back to issues
#52539·Docker

`docker system df` shows 265.5GB used for images without any images

Author: pjonssonCreated May 5, 2026Updated Sep 25, 2026
Labelsarea/builderarea/builder/buildkit

Description

Last week some nightly jobs failed to push to the registry with some error message about "unknown blob". Searching a bit led me to some page claiming that was caused by a polluted Docker cache, so I did docker system prune -af --volumes on 3 CI runners and ended up with this:

$ docker system df
TYPE            TOTAL     ACTIVE    SIZE      RECLAIMABLE
Images          0         0         265.5GB   0B (0%)
Containers      0         0         0B        0B
Local Volumes   0         0         0B        0B
Build Cache     0         0         0B        0B
$ docker system df -v
Images space usage:

REPOSITORY   TAG       IMAGE ID   CREATED   SIZE      SHARED SIZE   UNIQUE SIZE   CONTAINERS

Containers space usage:

CONTAINER ID   IMAGE     COMMAND   LOCAL VOLUMES   SIZE      CREATED   STATUS    NAMES

Local Volumes space usage:

VOLUME NAME   LINKS     SIZE

Build cache usage: 0B

CACHE ID   CACHE TYPE   SIZE      CREATED   LAST USED   USAGE     SHARED
$  ctr --debug images list
REF TYPE DIGEST SIZE PLATFORMS LABELS
$ sudo du -hxs /var/lib/containerd/
195G    /var/lib/containerd/

The exact numbers obviously differ between the machines, but same symptom, lots of space occupied but nothing that occupies it according to the Docker commands. I tried rebooting the machines, but that made no difference, it's still 265.5GB used without any images. The only way I could "fix" the situation was to purge the Docker packages, rm -rf /var/lib/containerd /var/lib/docker and reinstall the packages.

The Docker packages are purged the 15th every month followed by rm -rf /var/lib/docker and then the packages are reinstalled. I wasn't aware that containerd would also grow endlessly, so that directory had been growing since Docker switched the default to using containerd for image storage.

The output here comes from a paused runner that I will have to purge on Monday next week so it can resume its service.

Reproduce

I suspect this is the old bug where you build images concurrently and that gradually fills up /var/lib/docker/overlay2, but now it's with containerd as storage instead.

Expected behavior

If the reported size of images is >0, I would expect some image name or hash to be shown somewhere.

docker version

bash
Client: Docker Engine - Community
 Version:           29.4.2
 API version:       1.54
 Go version:        go1.26.2
 Git commit:        055a478
 Built:             Fri May  1 10:25:23 2026
 OS/Arch:           linux/amd64
 Context:           default

Server: Docker Engine - Community
 Engine:
  Version:          29.4.2
  API version:      1.54 (minimum version 1.40)
  Go version:       go1.26.2
  Git commit:       d329809
  Built:            Fri May  1 10:25:23 2026
  OS/Arch:          linux/amd64
  Experimental:     false
 containerd:
  Version:          v2.2.3
  GitCommit:        77c84241c7cbdd9b4eca2591793e3d4f4317c590
 runc:
  Version:          1.3.5
  GitCommit:        v1.3.5-0-g488fc13e
 docker-init:
  Version:          0.19.0
  GitCommit:        de40ad0

docker info

bash
Client: Docker Engine - Community
 Version:    29.4.2
 Context:    default
 Debug Mode: false
 Plugins:
  buildx: Docker Buildx (Docker Inc.)
    Version:  v0.33.0
    Path:     /usr/libexec/docker/cli-plugins/docker-buildx
  compose: Docker Compose (Docker Inc.)
    Version:  v5.1.3
    Path:     /usr/libexec/docker/cli-plugins/docker-compose

Server:
 Containers: 0
  Running: 0
  Paused: 0
  Stopped: 0
 Images: 0
 Server Version: 29.4.2
 Storage Driver: overlayfs
  driver-type: io.containerd.snapshotter.v1
 Logging Driver: json-file
 Cgroup Driver: systemd
 Cgroup Version: 2
 Plugins:
  Volume: local
  Network: bridge host ipvlan macvlan null overlay
  Log: awslogs fluentd gcplogs gelf journald json-file local splunk syslog
 CDI spec directories:
  /etc/cdi
  /var/run/cdi
 Swarm: inactive
 Runtimes: io.containerd.runc.v2 runc
 Default Runtime: runc
 Init Binary: docker-init
 containerd version: 77c84241c7cbdd9b4eca2591793e3d4f4317c590
 runc version: v1.3.5-0-g488fc13e
 init version: de40ad0
 Security Options:
  apparmor
  seccomp
   Profile: builtin
  cgroupns
 Kernel Version: 5.15.0-177-generic
 Operating System: Ubuntu 22.04.5 LTS
 OSType: linux
 Architecture: x86_64
 CPUs: 8
 Total Memory: 15.61GiB
 Name: ci-runner-medium-1
 ID: 13a934f1-c239-4a68-bee2-c3c6517b3d88
 Docker Root Dir: /var/lib/docker
 Debug Mode: false
 Experimental: false
 Insecure Registries:
  ::1/128
  127.0.0.0/8
 Registry Mirrors:
  https://mirror.here/
 Live Restore Enabled: false
 Firewall Backend: iptables

Additional Info

I'm not familiar with the internals of Docker and race conditions are always tricky, but it's possible #52538 that @vvoland just filed is another symptom related to this issue.

Source: moby/moby

View original on GitHubView discussion on GitHub