DinD cgroupv2 problem inside K8s

Description

Let's assume you have:

• cgroupv2 only OS (e.g. Ubuntu 22.04 or gardenlinux 934.7)
• A running Kubernetes cluster (e.g. Kind) on that OS
• Create a DinD pod with memory limits

Then create a container inside this DinD pod and it can consume more memory than the actual DinD pod is allowed to have...

Reproduce

Make sure your K8s node OS is cgroupv2 only:

mount | grep cgroup2

should return: cgroup2 on /sys/fs/cgroup type cgroup2 (rw,nosuid,nodev,noexec,relatime)

Then deploy Dind-pod to your Kluster:

apiVersion: v1
kind: Pod
metadata:
    name: dind
spec:
    containers:
      - name: dind
        image: docker:dind
        resources:
            requests:
                cpu: 10m
                memory: 300Mi
            limits:
                cpu: 2
                memory: 300Mi
        securityContext:
            privileged: true
            allowPrivilegeEscalation: true
        volumeMounts:
          - name: docker-graph-storage
            mountPath: /var/lib/docker
    volumes:
      - name: docker-graph-storage
        emptyDir: {}     

Then start some container inside the dind pod and consume more than 300 MB:

kubectl exec -it dind -- sh
docker run -it ubuntu bash
cat /dev/zero | head -c 400000000 | tail

Expected behavior

Process should have been killed. Also K8s metrics server is reporting wrong numbers. On Ubuntu 20 everything works as expected.

docker version

fails also with latest Docker version

docker info

see description

Additional Info

This "DinD in K8s" approach is a typical use case for self-hosted Github Runners on K8s with the Actions Runner Controller: https://github.com/actions/actions-runner-controller/issues/2284 We use it in our pipeline to spin up ephemeral Kind clusters which are running in the runner+dind pod.