external-snapshotter

CSI Snapshotter

The CSI snapshotter is part of Kubernetes implementation of Container Storage Interface (CSI) and implements both the volume snapshot and the volume group snapshot feature.

The volume snapshot feature supports CSI v1.0 and higher. It was introduced as an Alpha feature in Kubernetes v1.12 and has been promoted to a Beta feature in Kubernetes 1.17. In Kubernetes 1.20, the volume snapshot feature moves to GA.

The volume group snapshot feature supports CSI v1.10.0 and higher, and have been introduced in Kubernetes 1.27 as an alpha feature.

Overview

With the promotion of Volume Snapshot to GA, the feature is enabled by default on standard Kubernetes deployments and cannot be turned off.

The Volume Group Snapshot feature is disabled by default, and can be enabled using a command line option. For more information on how to enable it, see the usage section of this guide.

Blog post for the Snapshot GA feature can be found here.

Blog post for the Volume Group Snapshot Alpha feature can be found here.

Compatibility

This information reflects the head of this branch.

Note: snapshot-controller, snapshot-conversion-webhook, csi-snapshotter v4.1 requires v1 snapshot CRDs to be installed, but it serves both v1 and v1beta1 snapshot objects. Storage version is changed from v1beta1 to v1 in 4.1.0 so v1beta1 is deprecated and will be removed in a future release.

Note: when the volume group snapshot feature is enabled, snapshot-controller, snapshot-conversion-webhook, csi-snapshotter require the v1beta2 volumegroupsnapshot CRDs to be installed.

Feature Status

The VolumeSnapshotDataSource feature gate was introduced in Kubernetes 1.12 and it is enabled by default in Kubernetes 1.17 when the volume snapshot feature is promoted to beta. In Kubernetes 1.20, the feature gate is enabled by default on standard Kubernetes deployments and cannot be turned off.

Design

Both the snapshot controller and CSI external-snapshotter sidecar follow controller pattern and uses informers to watch for events. The snapshot controller watches for VolumeSnapshot and VolumeSnapshotContent create/update/delete events. Similarly, when the volume group snapshot feature is enabled, the snapshot controller watches for VolumeGroupSnapshot and VolumeGroupSnapshotContent create/update/delete events.

The CSI external-snapshotter sidecar only watches for VolumeSnapshotContent create/update/delete events. It filters out these objects with Driver== specified in the associated VolumeSnapshotClass object and then processes these events in workqueues with exponential backoff. Similarly, when the volume group snapshot feature is enabled, the CSI external-controller sidecar only watches for VolumeGroupSnapshotContent create/update/delete events.

The CSI external-snapshotter sidecar talks to CSI over socket (/run/csi/socket by default, configurable by -csi-address).

Snapshot v1 APIs

In the current release, both v1 and v1beta1 APIs are served while the stored API version is changed from v1beta1 to v1. v1beta1 APIs is deprecated and will be removed in a future release. It is recommended for users to switch to v1 APIs as soon as possible. Any previously created invalid v1beta1 objects have to be deleted before upgrading to version 4.1.

Volume Group Snapshot v1beta2 APIs

When enabled, the VolumeGroupSnapshot v1beta2 APIs are being served.

Usage

Volume Snapshot feature contains the following components:

• Kubernetes Volume Snapshot and Volume Group Snapshot CRDs
• Volume snapshot and volume group snapshot controller
• Volume group snapshot conversion webhook
• CSI Driver along with CSI Snapshotter sidecar

The Volume Snapshot feature depends on a volume snapshot controller and the volume snapshot CRDs. Both the controller and the CRDs are independent of any CSI driver. The CSI Snapshotter sidecar must run once per CSI driver. The single snapshot controller deployment works for all CSI drivers in a cluster. With leader election configured, the CSI sidecars and snapshot controller elect one leader per deployment. If deployed with two or more pods and leader election is enabled, the non-leader containers will attempt to get the lease. If the leader container dies, a non-leader will take over.

Therefore, it is strongly recommended that Kubernetes distributors bundle and deploy the controller and CRDs as part of their Kubernetes cluster management process (independent of any CSI Driver).

If your Kubernetes distribution does not bundle the snapshot controller, you may manually install these components by executing the following steps. Note that the snapshot controller YAML files in the git repository deploy into the default namespace for system testing purposes. For general use, update the snapshot controller YAMLs with an appropriate namespace prior to installing. For example, on a Vanilla Kubernetes cluster update the namespace from 'default' to 'kube-system' prior to issuing the kubectl create command.

There is a new conversion webhook server which provides conversion between v1beta1 and v1beta2 group snapshot objects. The cluster admin or Kubernetes distribution admin should install the webhook alongside the snapshot controllers and CRDs if they want to provide group snapshot v1beta1 API. More details below.).

Install Snapshot and Volume Group Snapshot CRDs:

• With the repo cloned locally: kubectl kustomize client/config/crd | kubectl create -f -
• From the repo remotely: kubectl kustomize https://github.com/kubernetes-csi/external-snapshotter/client/config/crd | kubectl create -f -
• Do this once per cluster

Install Common Snapshot Controller:

• Update the namespace to an appropriate value for your environment (e.g. kube-system)
• kubectl -n kube-system kustomize deploy/kubernetes/snapshot-controller | kubectl create -f -
• Do this once per cluster

Install CSI Driver:

• Follow instructions provided by your CSI Driver vendor.
• Here is an example to install the sample hostpath CSI driver
  • With the repo cloned locally: kubectl kustomize deploy/kubernetes/csi-snapshotter | kubectl create -f -
  • From the repo remotely: kubectl kustomize https://github.com/kubernetes-csi/external-snapshotter/deploy/kubernetes/csi-snapshotter | kubectl create -f -

Conversion Webhook

The snapshot conversion webhook is an HTTP callback which responds to conversion requests, allowing the API server to convert between the VolumeGroupSnapshotContent v1beta1 API to and from the v1beta2 API.

Read more about how to install the example webhook here.

Conversion Webhook Command Line Options

• --tls-cert-file: File containing the x509 Certificate for HTTPS. (CA cert, if any, concatenated after server cert). Required.

• --tls-private-key-file: File containing the x509 private key matching --tls-cert-file. Required.

• --port: Secure port that the webhook listens on (default 443)

Distributed Snapshotting

The distributed snapshotting feature is provided to handle snapshot operations for local volumes. To use this functionality, the snapshotter sidecar should be deployed along with the csi driver on each node so that every node manages the snapshot operations only for the volumes local to that node. This feature can be enabled by setting the following command line options to true:

Snapshot controller option

• --enable-distributed-snapshotting: This option lets the snapshot controller know that distributed snapshotting is enabled and the snapshotter sidecar will be running on each node. Off by default.

CSI external snapshotter sidecar option

• --node-deployment: Enables the snapshotter sidecar to handle snapshot operations for the volumes local to the node on which it is deployed. Off by default.

Other than this, the NODE_NAME environment variable must be set where the CSI snapshotter sidecar is deployed. The value of NODE_NAME should be the name of the node where the sidecar is running.

Volume Group Snapshot Support

The CSIVolumeGroupSnapshot feature gate is General Availability (GA) and enabled by default.

To use the volume group snapshot feature, install the Volume Group Snapshot CRDs in the cluster.

Note: Because the feature gate is enabled by default, both the snapshot controller and the CSI external-snapshotter sidecar check for the Volume Group Snapshot CRDs at startup. If those CRDs are not found, they log a warning and continue running with volume group snapshot support disabled for that process, rather than failing to start. This applies whether the feature gate is left at its default or explicitly set to true -- a CSI driver vendor or cluster admin has no control over whether a given cluster has the Volume Group Snapshot CRDs installed, so a missing CRD is never treated as a fatal startup error. To use volume group snapshots, install the CRDs; to silence the warning when you don't intend to use the feature, pass --feature-gates=CSIVolumeGroupSnapshot=false.

Snapshot controller command line options

Important optional arguments that are highly recommended to be used

• --leader-election: Enables leader election. This is useful when there are multiple replicas of the same snapshot controller running for the same Kubernetes deployment. Only one of them may be active (=leader). A new leader will be re-elected when current leader dies or becomes unresponsive for ~15 seconds.

• --leader-election-namespace : The namespace where the leader elect