S3 driver PurgeUploads causes extreme allocation activity in directoryDiff method

Author: MohanValliveduCreated Sep 4, 2026Updated Sep 4, 2026

Description

We are running Distribution Registry v3.1.1 with the S3 storage driver (s3-aws) as a Windows service written in Go. The registry uses an S3-compatible endpoint backed by a NAS environment using QuObjects. The Windows service has experienced repeated crashes and automatic restarts. Go heap profiling shows that the background PurgeUploads operation can generate extremely high allocation activity while walking the S3 storage.

Environment:

Distribution Registry: v3.1.1
Storage driver: S3 (s3-aws)
Platform: Windows
Go: 1.26.7
Object storage: QuObjects S3-compatible endpoint backed by NAS
Approximate number of objects in bucket: 2683 (~2 GB)

Observed behavior

During a problematic run, a heap profile showed approximately 28 GB of cumulative allocation, with the S3 directoryDiff function responsible for approximately 15.41 GiB, or 99.68% of the 15.46 GiB

Relevant profile:

Powershell statistics while process is running:

Time	PID	Working Set (MB)	Private (MB)	Virtual (GB)	Threads	System Commit (GB)	Commit Limit (GB)
02:37:08	3572	122.5	120.7	5.57	21	7.44	32.78
02:37:22	3572	502.6	501.4	5.85	21	8.23	32.78
02:37:58	3572	11,689.7	11,710.3	16.81	21	18.81	32.78
...	...	...	...	...	...	...	...
02:38:20	3572	11,670.0	24,022.2	29.19	21	30.76	34.80
...	...	...	...	...	...	...	...
02:39:41	3572	13,319.1	35,455.8	39.99	21	41.97	43.06
02:40:35	3572	12,243.9	34,586.3	40.03	21	41.18	44.07
02:40:36	3572	12,730.6	35,454.4	40.04	21	42.06	45.40

GO pprof tool shows:

> go tool pprof -list=directoryDiff heap.high
Total: 15.46GB
ROUTINE ======================== github.com/distribution/distribution/v3/registry/storage/driver/s3-aws.directoryDiff in /root/go/pkg/mod/github.com/distribution/distribution/[email protected]/registry/storage/driver/s3-aws/s3.go
   15.41GB    15.41GB (flat, cum) 99.68% of Total
         .          .   1243:func directoryDiff(prev, current string) []string {
         .          .   1244:   var paths []string
         .          .   1245:
         .          .   1246:   if prev == "" || current == "" {
         .          .   1247:           return paths
         .          .   1248:   }
         .          .   1249:
         .          .   1250:   parent := current
         .          .   1251:   for {
         .          .   1252:           parent = filepath.Dir(parent)
         .          .   1253:           if parent == "/" || parent == prev || strings.HasPrefix(prev+"/", parent+"/") {
         .          .   1254:                   break
         .          .   1255:           }
   15.41GB    15.41GB   1256:           paths = append(paths, parent)
         .          .   1257:   }
         .          .   1258:   slices.Reverse(paths)
         .          .   1259:   return paths
         .          .   1260:}
         .          .   1261:

The allocation tree shows the complete call chain:

startUploadPurger
  -> PurgeUploads
    -> getOutstandingUploads
      -> Base.Walk
        -> s3-aws.(*driver).Walk
          -> s3-aws.(*driver).doWalk
            -> s3.ListObjectsV2PagesWithContext
              -> s3-aws.(*driver).doWalk.func1
                -> directoryDiff

15.41 GiB (100%)  github.com/distribution/distribution/v3/registry/handlers.startUploadPurger.func1
					  app.go:1092
  -> 15.41 GiB (100%)  github.com/distribution/distribution/v3/registry/storage.PurgeUploads
						  purgeuploads.go:34
	-> 15.41 GiB (100%)  github.com/distribution/distribution/v3/registry/storage.getOutstandingUploads
							  purgeuploads.go:70
	  -> 15.41 GiB (100%)  github.com/distribution/distribution/v3/registry/storage/driver/base.(*Base).Walk
								  base.go:331
		-> 15.41 GiB (100%)  github.com/distribution/distribution/v3/registry/storage/driver/s3-aws.(*driver).Walk
									s3.go:1105
		  -> 15.41 GiB (100%)  github.com/distribution/distribution/v3/registry/storage/driver/s3-aws.(*driver).doWalk
									  s3.go:1151
			-> 15.41 GiB (100%)  github.com/aws/aws-sdk-go/service/s3.(*S3).ListObjectsV2PagesWithContext
										api.go:7654
			  -> 15.41 GiB (100%)  github.com/distribution/distribution/v3/registry/storage/driver/s3-aws.(*driver).doWalk.func1
										  s3.go:1158
				-> 15.41 GiB (100%)  github.com/distribution/distribution/v3/registry/storage/driver/s3-aws.directoryDiff
											  s3.go:1256  paths = append(paths, parent)

After the Windows service was restarted, a profile from a healthy process showed only approximately 15.95 MiB of live heap and no significant directoryDiff allocation. A separate healthy-process alloc_space capture showed approximately 241 MB of cumulative allocation. This suggests that the excessive state is process-local and associated with the S3 walk.

Workaround

We disabled the registry purging option. With purging disabled, the observed allocation spike does not occur.

Reproduce

N/A

Expected behavior

No response

registry version

go.mod github.com/distribution/distribution/v3 v3.1.1

Additional Info

No response

Source: distribution/distribution

View original on GitHub View discussion on GitHub