Kopia S3 connections leak in long-running Velero server process

What happened

The Velero server process accumulates TCP connections to S3 in ESTABLISHED state over time when using kopia data mover (snapshotMoveData: true). These connections are never cleaned up. A user reported 5,775 open connections from a single Velero pod after one week, with no active backups or restores running at the time.

Restarting the Velero pod temporarily clears the connections, but they accumulate again.

Why this happens

Every time Velero opens a kopia repository (for a data mover upload, download, or maintenance run), it creates a new cloned http.Transport in getCustomTransport():

transport := http.DefaultTransport.(*http.Transport).Clone()

When the repo is closed after the operation finishes, s3Storage.Close() is inherited from blob.DefaultProviderImplementation, which is a no-op. The transport and its connection pool are never cleaned up.

Normally you'd expect Go's GC to handle this, but http.Transport is never garbage collected while it holds idle connections. The idle connection goroutines hold references back to the transport, preventing collection.

The transport does have IdleConnTimeout: 90s from DefaultTransport, but connections that get reused within that window (e.g., by BSL validation running every 60s) never go idle long enough to be cleaned up.

With 4 backups/day plus hourly kopia maintenance, that's roughly 28 new orphaned transports per day, each with their own connection pool. Over a week, that adds up.

User impact

The user has an on-prem S3-compatible store behind a NetScaler load balancer. The NetScaler tracks state for every TCP connection. Thousands of leaked connections from Velero pods across multiple clusters caused memory exhaustion on the NetScaler, making S3 unavailable.

Reported environment

• Velero 1.14.x (via OADP 1.4.8 and 1.4.9)
• Kubernetes/OCP 4.16, 4.18, 4.20
• MinIO behind NetScaler load balancer
• Kopia uploader with snapshotMoveData: true
• Reproducible across multiple clusters

This likely affects all Velero versions that use the kopia uploader (1.10+, default since 1.12), as the underlying kopia S3 storage has never implemented Close() cleanup.

How to check

From inside the Velero pod:

cat /proc/net/tcp | awk '{print $4}' | sort | uniq -c | sort -rn | head

State 01 = ESTABLISHED. The count grows over time and never decreases.

Suggested fix

Have s3Storage store a reference to the transport and override Close():

func (s *s3Storage) Close(ctx context.Context) error {
    if s.transport != nil {
        s.transport.CloseIdleConnections()
    }
    return nil
}

The fix would go in project-velero/kopia. Upstream kopia/kopia has the same gap but it's less of a problem there since kopia CLI runs as a short-lived process.