Kubernetes Operator for Zookeeper
The project is currently alpha. While no breaking API changes are currently planned, we reserve the right to address bugs and change the API before the project is declared stable.
This operator runs a Zookeeper 3.9.3 cluster, and uses Zookeeper dynamic reconfiguration to handle node membership.
The operator itself is built with the Operator framework.
We recommend using our helm charts for all installation and upgrades. Since version 0.2.8 onwards, the helm charts for zookeeper operator and zookeeper cluster are published in https://charts.pravega.io. To add this repository to your Helm repos, use the following command
helm repo add pravega https://charts.pravega.ioHowever there are manual deployment and upgrade options available as well.
Note: if you are running on Google Kubernetes Engine (GKE), please check this first.
To understand how to deploy the zookeeper operator using helm, refer to this.
Register the ZookeeperCluster custom resource definition (CRD).
$ kubectl create -f config/crd/basesYou can choose to enable Zookeeper operator for all namespaces or just for a specific namespace. The example is using the default namespace, but feel free to edit the Yaml files and use a different namespace.
Create the operator role and role binding.
// default namespace
$ kubectl create -f config/rbac/default_ns_rbac.yaml
// all namespaces
$ kubectl create -f config/rbac/all_ns_rbac.yamlDeploy the Zookeeper operator.
$ kubectl create -f config/manager/manager.yamlVerify that the Zookeeper operator is running.
$ kubectl get deploy
NAME DESIRED CURRENT UP-TO-DATE AVAILABLE AGE
zookeeper-operator 1 1 1 1 12mTo understand how to deploy a sample zookeeper cluster using helm, refer to this.
Create a Yaml file called zk.yaml with the following content to install a 3-node Zookeeper cluster.
apiVersion: "zookeeper.pravega.io/v1beta1"
kind: "ZookeeperCluster"
metadata:
name: "zookeeper"
spec:
replicas: 3$ kubectl create -f zk.yamlAfter a couple of minutes, all cluster members should become ready.
$ kubectl get zk
NAME REPLICAS READY REPLICAS VERSION DESIRED VERSION INTERNAL ENDPOINT EXTERNAL ENDPOINT AGE
zookeeper 3 3 0.2.8 0.2.8 10.100.200.18:2181 N/A 94sNote: when the Version field is set as well as Ready Replicas are equal to Replicas that signifies our cluster is in Ready state
Additionally, check the output of describe command which should show the following cluster condition
$ kubectl describe zk
Conditions:
Last Transition Time: 2020-05-18T10:17:03Z
Last Update Time: 2020-05-18T10:17:03Z
Status: True
Type: PodsReady
Note: User should wait for the Pods Ready condition to be True
…Note: If you want to configure zookeeper pod, for example to change the service account or the CPU limits, you can set the following properties: ~/charts/zookeeper/templates/zookeeper.yaml. Service account configuration is available from zookeeper operator version
0.2.9onwards.
apiVersion: "zookeeper.pravega.io/v1beta1"
kind: "ZookeeperCluster"
metadata:
name: "example"
spec:
pod:
serviceAccountName: "zookeeper"
resources:
requests:
cpu: 200m
memory: 256Mi
limits:
cpu: 200m
memory: 256MiCreate a Yaml file called zk.yaml with the following content to install a 3-node Zookeeper cluster.
apiVersion: "zookeeper.pravega.io/v1beta1"
kind: "ZookeeperCluster"
metadata:
name: "example"
spec:
replicas: 3
storageType: ephemeral$ kubectl create -f zk.yamlAfter a couple of minutes, all cluster members should become ready.
$ kubectl get zk
NAME REPLICAS READY REPLICAS VERSION DESIRED VERSION INTERNAL ENDPOINT EXTERNAL ENDPOINT AGE
example 3 3 0.2.7 0.2.7 10.100.200.18:2181 N/A 94sNote: User should only provide value for either the field persistence or ephemeral in the spec if none of the values is specified default is persistence
Note: In case of ephemeral storage, the cluster may not be able to come back up if more than quorum number of nodes are restarted simultaneously.
Note: In case of ephemeral storage, there will be loss of data when the node gets restarted.
Create a Yaml file called zk-with-istio.yaml with the following content to install a 3-node Zookeeper cluster.
apiVersion: zookeeper.pravega.io/v1beta1
kind: ZookeeperCluster
metadata:
name: zk-with-istio
spec:
replicas: 3
config:
initLimit: 10
tickTime: 2000
syncLimit: 5
quorumListenOnAllIPs: true$ kubectl create -f zk-with-istio.yamlTo understand how to upgrade the zookeeper cluster using helm, refer to this.
To initiate an upgrade process manually, a user has to update the spec.image.tag field of the ZookeeperCluster custom resource. This can be done in three different ways using the kubectl command.
kubectl edit zk , modify the tag value in the YAML resource, save, and exit.zk.yaml, you can modify the tag value, and reapply the resource with kubectl apply -f zk.yaml.kubectl patch zk --type='json' -p='[{"op": "replace", "path": "/spec/image/tag", "value": "X.Y.Z"}]'.After the tag field is updated, the StatefulSet will detect the version change and it will trigger the upgrade process.
To detect whether a ZookeeperCluster upgrade is in progress or not, check the output of the command kubectl describe zk. Output of this command should contain the following entries
$ kubectl describe zk
status:
Last Transition Time: 2020-05-18T10:25:12Z
Last Update Time: 2020-05-18T10:25:12Z
Message: 0
Reason: Updating Zookeeper
Status: True
Type: UpgradingAdditionally, the Desired Version will be set to the version that we are upgrading our cluster to.
$ kubectl get zk
NAME REPLICAS READY REPLICAS VERSION DESIRED VERSION INTERNAL ENDPOINT EXTERNAL ENDPOINT AGE
zookeeper 3 3 0.2.6 0.2.7 10.100.200.126:2181 N/A 11m
Once the upgrade completes, the Version field is set to the Desired Version, as shown below
$ kubectl get zk
NAME REPLICAS READY REPLICAS VERSION DESIRED VERSION INTERNAL ENDPOINT EXTERNAL ENDPOINT AGE
zookeeper 3 3 0.2.7 0.2.7 10.100.200.126:2181 N/A 11m
Additionally, the Upgrading status is set to False and PodsReady status is set to True, which signifies that the upgrade has completed, as shown below
$ kubectl describe zk
Status:
Conditions:
Last Transition Time: 2020-05-18T10:28:22Z
Last Update Time: 2020-05-18T10:28:22Z
Status: True
Type: PodsReady
Last Transition Time: 2020-05-18T10:28:22Z
Last Update Time: 2020-05-18T10:28:22Z
Status: False
Type: UpgradingNote: The value of the tag field should not be modified while an upgrade is already in progress.
For upgrading the zookeeper operator check the document operator-upgrade
Refer to this.
$ kubectl delete -f zk.yamlNote that the Zookeeper clusters managed by the Zookeeper operator will NOT be deleted even if the operator is uninstalled.
Refer to this.
To delete all clusters, delete all cluster CR objects before uninstalling the operator.
$ kubectl delete -f config/manager/manager.yaml
$ kubectl delete -f config/rbac/default_ns_rbac.yaml
// or, depending on how you deployed it
$ kubectl delete -f config/rbac/all_ns_rbac.yamlThe AdminServer is an embedded Jetty server that provides an HTTP interface to the four letter word commands. This port is made accessible to the outside world via the AdminServer service. By default, the server is started on port 8080, but this configuration can be modified by providing the desired port number within the values.yaml file of the zookeeper cluster charts
ports:
- containerPort: 8118
name: admin-serverThis would bring up the AdminServer service on port 8118 as shown below
$ kubectl get svc
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S)
zookeeper-admin-server LoadBalancer 10.100.200.104 10.243.39.62 8118:30477/TCPThe commands are issued by going to the URL /commands/, e.g. http://10.243.39.62:8118/commands/stat
The list of available commands are
/commands/configuration
/commands/connection_stat_reset
/commands/connections
/commands/dirs
/commands/dump
/commands/environment
/commands/get_trace_mask
/commands/hash
/commands/initial_configuration
/commands/is_read_only
/commands/last_snapshot
/commands/leader
/commands/monitor
/commands/observer_connection_stat_reset
/commands/observers
/commands/ruok
/commands/server_stats
/commands/set_trace_mask
/commands/stat_reset
/commands/stats
/commands/system_properties
/commands/voting_view
/commands/watch_summary
/commands/watches
/commands/watches_by_path
/commands/zabstateRequirements:
Use the make command to build the Zookeeper operator im
No open issues yet, or sync has not completed.