Shim is sometimes not cleaned up by StopPodSandbox

Description

While running the integration test suite, I found that TestContainerdRestart leaks a shim process about 30% of the time. I was able to narrow down the cause to the task which is created for the notready-sandbox exited-container container. Removing the unrelated parts from the TestContainerdRestart test, the leak can be reproduced with the following test:

package integration

import (
    "context"
    "syscall"
    "testing"

    containerd "github.com/containerd/containerd/v2/client"
    "github.com/stretchr/testify/assert"
    runtime "k8s.io/cri-api/pkg/apis/runtime/v1"
)

func TestShimFileLeakReproducer(t *testing.T) {
    ctx := context.Background()

    sbCfg := PodSandboxConfig("test-sandbox", "test-shim-file-leak-reproducer")
    sid, err := runtimeService.RunPodSandbox(sbCfg, "")
    assert.NoError(t, err)

    cfg := ContainerConfig(
        "test-container", "registry.k8s.io/pause:3.10", 
        WithPidNamespace(runtime.NamespaceMode_CONTAINER),
    )
    cid, err := runtimeService.CreateContainer(sid, cfg, sbCfg)
    assert.NoError(t, err)

    err = runtimeService.StartContainer(cid)
    assert.NoError(t, err)

    cntr, err := containerdClient.LoadContainer(ctx, sid)
    assert.NoError(t, err)
    task, err := cntr.Task(ctx, nil)
    assert.NoError(t, err)

    err = task.Kill(ctx, syscall.SIGKILL, containerd.WithKillAll)
    assert.NoError(t, err)

    assert.NoError(t, runtimeService.StopPodSandbox(sid))
    assert.NoError(t, runtimeService.RemovePodSandbox(sid))
}

The process which is leaked is created by the core/runtime/v2/binary/binary.Start method when it runs the command /usr/local/bin/containerd-shim-runc-v2 -namespace k8s.io -address /run/containerd/containerd.sock -publish-binary /usr/local/bin/containerd -id <container-id> start. The leak only happens when the container task is killed. Adding delays before and after killing the task does not seem to affect how often the leak happens.

Based on an offline discussion with @samuelkarp, it seems that this could be an issue with runtimeService.StopPodSandbox not properly cleaning up the shim when trying to clean up the pod resources, and doesn't seem specific to the test code.

Steps to reproduce the issue

1. Run go test -run '^TestContainerdRestart$' or go test -run TestShimFileLeakReproducer several times
2. Check pgrep -a containerd-shim and observe that processes have leaked

Describe the results you received and expected

Results

After running StopPodSandbox(sid), a process like this is sometimes still present: 1537377 /usr/local/bin/containerd-shim-runc-v2 -namespace k8s.io -id <sid> -address /run/containerd/containerd.sock.

Expectation

After running StopPodSandbox(sid), all related processes should be cleaned up.

What version of containerd are you using?

master

Any other relevant information

$ uname -a Linux kpberry-containerd-test 6.1.0-21-cloud-amd64 #1 SMP PREEMPT_DYNAMIC Debian 6.1.90-1 (2024-05-03) x86_64 GNU/Linux

Show configuration if it is related to CRI plugin.

No response