Hub upgrade deadlocks on ReadWriteOnce snapshots PVC: Deployment needs strategy: Recreate

Summary

When tap.snapshots.local.storageClass is set, the hub Deployment mounts a ReadWriteOnce PVC but uses the default RollingUpdate strategy. Any hub upgrade where the new pod is scheduled onto a different node than the old one deadlocks: the new pod cannot attach the volume, and RollingUpdate will not terminate the old pod until the new one is Ready.

Neither side can make progress. It requires manual intervention to clear.

What happens

Warning  FailedAttachVolume  attachdetach-controller
  Multi-Attach error for volume "pvc-3d23b4f0-3bcc-40d2-9650-c3f3dfae22fb"
  Volume is already used by pod(s) kubeshark-hub-8486b5fdcb-kzzds

The new hub pod sat in ContainerCreating for ~12 minutes until the old pod was deleted by hand. It would have stayed there indefinitely.

NAME                              READY   STATUS              AGE
kubeshark-hub-65bbcf56bf-jc8gt    0/1     ContainerCreating   12m   <- new, cannot attach
kubeshark-hub-8486b5fdcb-kzzds    1/1     Running             12h   <- old, holds the volume

Why it is structural, not environmental

Three chart facts combine:

1. templates/09-snapshots-pvc.yaml hardcodes the access mode:
   yaml复制

   spec:
     accessModes:
       - ReadWriteOnce

2. templates/04-hub-deployment.yaml (~L222) mounts it whenever a storage class is configured:
   yaml复制

   - name: snapshots-volume
     {{- if .Values.tap.snapshots.local.storageClass }}
     persistentVolumeClaim:
       claimName: {{ include "kubeshark.name" . }}-snapshots-pvc
     {{- else }}
     emptyDir:

3. templates/04-hub-deployment.yaml spec: (L14) sets no strategy, so Kubernetes defaults to RollingUpdate with maxSurge: 25% → the new pod is created before the old one is removed.

A ReadWriteOnce volume (EBS and most block storage) can only attach to one node at a time. With a multi-node cluster the scheduler will regularly place the new pod elsewhere, so this is expected to hit any multi-node user who persists snapshots — not a rare race.

emptyDir users are unaffected, which is probably why it has gone unnoticed.

The failure is silent

helm upgrade returns success and reports STATUS: deployed while the hub is wedged, because the manifests applied cleanly. Unless you check pod status afterwards, the upgrade looks fine — meanwhile the hub serving traffic is still the old build, so a "completed" upgrade silently keeps running the previous version.

Suggested fix

Use Recreate for the hub Deployment, at least when the PVC is in play:

spec:
  replicas: 1
  strategy:
    type: Recreate

replicas is already hardcoded to 1, so there is no availability benefit from RollingUpdate here — it only introduces the overlap that causes the deadlock. Recreate accepts a few seconds of hub downtime during upgrade, which is what effectively happens anyway.

If a brief gap is unacceptable, gating it works too:

{{- if .Values.tap.snapshots.local.storageClass }}
  strategy:
    type: Recreate
{{- end }}

RollingUpdate with maxSurge: 0 would also avoid the overlap, but Recreate states the intent more plainly for a single-replica, single-writer component.

Workaround

Delete the old hub pod to release the volume; the new one then attaches and starts:

kubectl -n <ns> delete pod <old-hub-pod>

Environment

• Chart from master (helm-chart/), Chart.yaml version 53.3.0
• EKS 1.36, 3 nodes, aws-ebs-csi-driver v1.63.1
• tap.snapshots.local.storageClass: gp2, storageSize: 100Gi
• Reproduced upgrading hub docker.io/kubeshark/hub:v53.3 → a master build