Performance Degradation for Namespace Deletion from 1.34.4 -> 1.34.5 and 1.35.1 -> 1.35.2
Environmental Info: K3s Version: 1.34.5, 1.35.2
Node(s) CPU architecture, OS, and Version:
Run in Docker
Cluster Configuration:
1 server, 0 agents
Describe the bug:
Namespace deletion seems to be taking longer after the specific version transitions:
- 1.34.4 -> 1.34.5
- 1.35.1 -> 1.35.2
We use k3s via k3d as part of our testing framework and pipelines, so fast namespace deletion when required is the most useful to reduce testing time.
Within our testing framework, we are seeing increases of up to 30 seconds -> 1 minute for namespaces to be deleted, despite all resources within the cluster already being gone. The namespace is mentioning PVCs, but I confirmed the PVCs were gone by then. I have not yet been able to reproduce this specific scenario in a lightweight k3s cluster, but I can confirm this new behavior was observed because of the version changes.
With a lightweight reproduction of just ConfigMaps + Secrets in a namespace, it goes up from 5 seconds to 10 seconds. This is consistent in timing.
Steps To Reproduce:
We use k3s via k3d.
The script used to create the clusters, which is very close to how we set up our clusters for testing:
#!/usr/bin/env bash
set -euo pipefail
K8S_VERSION="${1:?Usage: $0 <k8s-version> (e.g. 1.31.0)}"
SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
ARCH="$(uname -m)"
[[ "$ARCH" == "x86_64" ]] && ARCH=amd64
[[ "$ARCH" == "aarch64" ]] && ARCH=arm64
K3D="${K3D:-k3d}"
K3D_CLUSTER_NAME="${K3D_CLUSTER_NAME:-test-${K8S_VERSION}}"
K3D_REGISTRIES_FILE="${SCRIPT_DIR}/registries.yaml"
K3D_CONFIG_FILE="${SCRIPT_DIR}/k3d-cluster.yaml"
K3D_KUBECONFIG_FILE="${SCRIPT_DIR}/kubeconfig.${K3D_CLUSTER_NAME}.yaml"
TEST_TMP_DIR="${TEST_TMP_DIR:-/tmp/k3s-testing}"
mkdir -p "${TEST_TMP_DIR}/kubernetes/audit"
rm -f "${TEST_TMP_DIR}/kubernetes/audit/audit.log"
touch "${TEST_TMP_DIR}/kubernetes/audit/audit.log"
chmod a+w "${TEST_TMP_DIR}/kubernetes/audit/audit.log"
while ! "${K3D}" cluster create "${K3D_CLUSTER_NAME}" \
--registry-config "${K3D_REGISTRIES_FILE}" \
--image "rancher/k3s:v${K8S_VERSION}-k3s1" \
--config "${K3D_CONFIG_FILE}" \
--volume "${SCRIPT_DIR}/audit-policy.yaml:/var/lib/rancher/k3s/server/manifests/audit-policy.yaml" \
--volume "${SCRIPT_DIR}/traefik-config.yaml:/var/lib/rancher/k3s/server/manifests/traefik-config.yaml" \
--volume "${TEST_TMP_DIR}/kubernetes:/var/log/kubernetes"; do
sleep 1
done
K3D_CLUSTER_HOST="k3d-${K3D_CLUSTER_NAME}-server-0"
K3D_CLUSTER_API="${K3D_CLUSTER_HOST}:6443"
"${K3D}" kubeconfig get "${K3D_CLUSTER_NAME}" \
| sed "s#0\.0\.0\.0:[0-9]*#${K3D_CLUSTER_API}#g" \
> "${K3D_KUBECONFIG_FILE}"
echo "Kubeconfig written to ${K3D_KUBECONFIG_FILE}"Relevant files:
audit-policy.yaml
apiVersion: audit.k8s.io/v1
kind: Policy
rules:
- level: None
namespaces: ["kube-system","kube-public","kube-node-lease","vault","api-mocks"]
- level: None
verbs: ["get","list","watch"]
- level: Metadatatraefik-config.yaml
apiVersion: helm.cattle.io/v1
kind: HelmChartConfig
metadata:
name: traefik
namespace: kube-system
spec:
valuesContent: |-
ingressClass:
enabled: true
name: haproxy-default
providers:
kubernetesIngress:
ingressClass: haproxy-default
deployment:
podLabels:
app: nginx-ingressregistries.yaml is omitted
The script used to test the namespace behavior:
#!/bin/bash
kubectl create namespace hello
kubectl create configmap -n hello test-config-map
kubectl create configmap -n hello test-config-map-2
kubectl create secret generic -n hello test-secret
kubectl create secret generic -n hello test-secret-2
date +"%T.%3N"
kubectl delete namespace hello
date +"%T.%3N"Expected behavior:
Namespace deletion should be near-immediate once all resources in the namespace are gone.
Actual behavior:
Longer periods of time for namespace deletion to complete even though all resources in the namespace are gone.
Additional context / logs:
Here are the patterns we saw, with only the k3s image changing each time:
Our direct test framework with the PVC behavior, with script adjustments to include the PVC changes:
1.34
1.34.4 -> good
./nsCreate.sh
namespace/hello created
persistentvolumeclaim/guest-vol1 created
23:51:41.094
Waiting for PVC 'guest-vol1' in namespace 'hello' to be Bound...
PVC 'guest-vol1' is Bound.
23:52:11.541
namespace "hello" deleted
23:52:22.755
1.34.5 -> bad
./nsCreate.sh
namespace/hello created
persistentvolumeclaim/guest-vol1 created
23:57:51.842
Waiting for PVC 'guest-vol1' in namespace 'hello' to be Bound...
PVC 'guest-vol1' is Bound.
23:58:01.825
namespace "hello" deleted
23:58:44.865
1.35
1.35.1 -> good
./nsCreate.sh
namespace/hello created
persistentvolumeclaim/guest-vol1 created
00:13:07.216
Waiting for PVC 'guest-vol1' in namespace 'hello' to be Bound...
PVC 'guest-vol1' is Bound.
00:13:10.682
namespace "hello" deleted
00:13:22.072
1.35.2 -> bad
./nsCreate.sh
namespace/hello created
persistentvolumeclaim/guest-vol1 created
00:18:33.103
Waiting for PVC 'guest-vol1' in namespace 'hello' to be Bound...
PVC 'guest-vol1' is Bound.
00:18:39.850
namespace "hello" deleted
00:19:39.276Lightweight namespace experiment:
1.34
1.34.4
./nsCreate.sh
namespace/hello created
configmap/test-config-map created
configmap/test-config-map-2 created
secret/test-secret created
secret/test-secret-2 created
11:41:56.239
namespace "hello" deleted
11:42:01.588
./nsCreate.sh
namespace/hello created
configmap/test-config-map created
configmap/test-config-map-2 created
secret/test-secret created
secret/test-secret-2 created
11:42:20.292
namespace "hello" deleted
11:42:25.642
1.34.5
./nsCreate.sh
namespace/hello created
configmap/test-config-map created
configmap/test-config-map-2 created
secret/test-secret created
secret/test-secret-2 created
11:43:54.000
namespace "hello" deleted
11:44:09.506
./nsCreate.sh
namespace/hello created
configmap/test-config-map created
configmap/test-config-map-2 created
secret/test-secret created
secret/test-secret-2 created
11:44:21.385
namespace "hello" deleted
11:44:37.595
1.35
1.35.1 - good
./nsCreate.sh
namespace/hello created
configmap/test-config-map created
configmap/test-config-map-2 created
secret/test-secret created
secret/test-secret-2 created
11:37:10.865
namespace "hello" deleted
11:37:16.238
./nsCreate.sh
namespace/hello created
configmap/test-config-map created
configmap/test-config-map-2 created
secret/test-secret created
secret/test-secret-2 created
11:37:34.139
namespace "hello" deleted
11:37:39.501
./nsCreate.sh
namespace/hello created
configmap/test-config-map created
configmap/test-config-map-2 created
secret/test-secret created
secret/test-secret-2 created
11:37:51.018
namespace "hello" deleted
11:37:56.378
1.35.2 - bad
./nsCreate.sh
namespace/hello created
configmap/test-config-map created
configmap/test-config-map-2 created
secret/test-secret created
secret/test-secret-2 created
11:39:10.302
namespace "hello" deleted
11:39:25.487
./nsCreate.sh
namespace/hello created
configmap/test-config-map created
configmap/test-config-map-2 created
secret/test-secret created
secret/test-secret-2 created
11:39:35.481
namespace "hello" deleted
11:39:50.531
./nsCreate.sh
namespace/hello created
configmap/test-config-map created
configmap/test-config-map-2 created
secret/test-secret created
secret/test-secret-2 created
11:39:59.395
namespace "hello" deleted
11:40:17.059Other context:
- This issue is still present in the latest corresponding version of 1.34 and 1.35 (1.34.9 and 1.35.6 respectively)
- I used AI to check the diff between the tags:
- https://github.com/k3s-io/k3s/compare/v1.34.4+k3s1...v1.34.5+k3s1
- https://github.com/k3s-io/k3s/compare/v1.35.1+k3s1...v1.35.2+k3s1
- It called out a change in kine from v0.14.11 to v0.14.12
- When looking at the diff in kine, it called out this specific change as potentially causing the new slowness - https://github.com/k3s-io/kine/pull/602
- Also noting that issue https://github.com/k3s-io/kine/issues/605 which mentions slowness of namespace deletion was opened the same day that change was merged
- but as the experts, I would like you to confirm this is the real issue
- As of now, we are able to make do with using 1.35.1 but we would have problems with 1.36+ images, which we would need to use in order to simulate our live environments
Source: k3s-io/k3s