runtime-rs: Lack of atomicity in the block device hotplugging operation causing "Duplicate nodes" errors.
Author: ApokleosCreated Aug 1, 2025Updated Sep 17, 2026
Labelsbugneeds-review
Get your issue reviewed faster
not ok 3 Test we can pull an image inside the guest using trusted storage
# (from function `k8s_create_pod' in file lib.sh, line 104,
# in test file k8s-guest-pull-image.bats, line 137)
# `k8s_create_pod "$pod_config" "$wait_time"' failed
# No resources found in kata-containers-k8s-tests namespace.
# [bats-exec-test:73] INFO: k8s configured to use runtimeclass
# 2500+0 records in
# 2500+0 records out
# 2621440000 bytes (2.6 GB, 2.4 GiB) copied, 7.43136 s, 353 MB/s
# Trusted storage /tmp/bats-run-0Eiz7v/file/0/trusted-storage.yaml.in.sNS file:
# #
# # Copyright (c) 2024 Intel Corporation
# #
# # SPDX-License-Identifier: Apache-2.0
# #
#
# apiVersion: storage.k8s.io/v1
# kind: StorageClass
# metadata:
# name: local-storage
# provisioner: kubernetes.io/no-provisioner
# volumeBindingMode: WaitForFirstConsumer
# ---
# apiVersion: v1
# kind: PersistentVolume
# metadata:
# name: trusted-block-pv
# spec:
# capacity:
# storage: 10Gi
# volumeMode: Block
# accessModes:
# - ReadWriteOnce
# persistentVolumeReclaimPolicy: Retain
# storageClassName: local-storage
# local:
# path: /dev/loop0
# nodeAffinity:
# required:
# nodeSelectorTerms:
# - matchExpressions:
# - key: kubernetes.io/hostname
# operator: In
# values:
# - aks-nodepool1-26885032-vmss000000
# ---
# apiVersion: v1
# kind: PersistentVolumeClaim
# metadata:
# name: trusted-pvc
# spec:
# accessModes:
# - ReadWriteOnce
# resources:
# requests:
# storage: 1Gi
# volumeMode: Block
# storageClassName: local-storage
# retry_kubectl_apply: Attempt 1/5
# storageclass.storage.k8s.io/local-storage created
# persistentvolume/trusted-block-pv created
# persistentvolumeclaim/trusted-pvc created
# retry_kubectl_apply: Resource created successfully.
# metadata.annotations."io.katacontainers.config.hypervisor.kernel_params"
# metadata.annotations."io.containerd.cri.runtime-handler"
# Pod /tmp/bats-run-0Eiz7v/file/0/pod-guest-pull-in-trusted-storage.yaml.in.7sa file:
# #
# # Copyright (c) 2024 Intel Corporation
# #
# # SPDX-License-Identifier: Apache-2.0
# #
# apiVersion: v1
# kind: Pod
# metadata:
# name: large-image-pod
# annotations:
# io.katacontainers.config.hypervisor.kernel_params: agent.secure_storage_integrity=true
# io.containerd.cri.runtime-handler: kata-qemu-runtime-rs-coco-dev
# spec:
# runtimeClassName: kata
# affinity:
# nodeAffinity:
# requiredDuringSchedulingIgnoredDuringExecution:
# nodeSelectorTerms:
# - matchExpressions:
# - key: kubernetes.io/hostname
# operator: In
# values:
# - aks-nodepool1-26885032-vmss000000
# volumes:
# - name: trusted-storage
# persistentVolumeClaim:
# claimName: trusted-pvc
# containers:
# - name: app-container
# image: ghcr.io/confidential-containers/test-container:rust-1.79.0
# command: ["/bin/sh", "-c"]
# args:
# - sleep 6000
# volumeDevices:
# - devicePath: /dev/trusted_store
# name: trusted-storage
# [bats-exec-test:310] INFO: Adding allow all policy to Pod from /tmp/bats-run-0Eiz7v/file/0/pod-guest-pull-in-trusted-storage.yaml.in.7sa
#
# retry_kubectl_apply: Attempt 1/5
# pod/large-image-pod created
# retry_kubectl_apply: Resource created successfully.
# error: timed out waiting for the condition on pods/large-image-pod
# NAME READY STATUS RESTARTS AGE
# large-image-pod 0/1 CrashLoopBackOff 3 (41s ago) 2m
# Name: large-image-pod
# Namespace: kata-containers-k8s-tests
# Priority: 0
# Runtime Class Name: kata
# Service Account: default
# Node: aks-nodepool1-26885032-vmss000000/10.224.0.4
# Start Time: Thu, 31 Jul 2025 09:50:39 +0000
# Labels: <none>
# Annotations: io.containerd.cri.runtime-handler: kata-qemu-runtime-rs-coco-dev
# io.katacontainers.config.agent.policy:
# IyBDb3B5cmlnaHQgKGMpIDIwMjMgTWljcm9zb2Z0IENvcnBvcmF0aW9uCiMKIyBTUERYLUxpY2Vuc2UtSWRlbnRpZmllcjogQXBhY2hlLTIuMAojCgpwYWNrYWdlIGFnZW50X3BvbG...
# io.katacontainers.config.hypervisor.kernel_params: agent.secure_storage_integrity=true
# Status: Running
# IP: 10.244.0.193
# IPs:
# IP: 10.244.0.193
# Containers:
# app-container:
# Container ID: containerd://88c910d6a72a26f8fb7f4fc8c6702ebe1ed3a5b86184bfd5d2c2d6270fa81ea2
# Image: ghcr.io/confidential-containers/test-container:rust-1.79.0
# Image ID: ghcr.io/confidential-containers/test-container@sha256:981b60160a09698712d30c238d870523dc5e94dfaaa0da18d9ba6fb5f4734552
# Port: <none>
# Host Port: <none>
# Command:
# /bin/sh
# -c
# Args:
# sleep 6000
# State: Waiting
# Reason: CrashLoopBackOff
# Last State: Terminated
# Reason: StartError
# Message: failed to create containerd task: failed to create shim task: Others("failed to handle message create container\n\nCaused by:\n 0: do handle device\n 1: failed to add deivce\n 2: hotplug block device\n 3: blockdev-add backend Qapi(Error { class: GenericError, desc: \"Duplicate nodes with node-name='drive-1'\", id: None })\n\nStack backtrace:\n 0: anyhow::error::<impl anyhow::Error>::msg\n 1: hypervisor::qemu::qmp::Qmp::hotplug_block_device\n 2: hypervisor::qemu::inner::QemuInner::hotplug_device\n 3: <hypervisor::qemu::Qemu as hypervisor::Hypervisor>::add_device::{{closure}}\n 4: <hypervisor::device::driver::virtio_blk::BlockDevice as hypervisor::device::Device>::attach::{{closure}}\n 5: hypervisor::device::device_manager::do_handle_device::{{closure}}\n 6: virt_container::container_manager::container::Container::create::{{closure}}\n 7: <virt_container::container_manager::manager::VirtContainerManager as common::container_manager::ContainerManager>::create_container::{{closure}}\n 8: runtimes::manager::RuntimeHandlerManager::handler_task_message::{{closure}}::{{closure}}\n 9: runtimes::manager::RuntimeHandlerManager::handler_task_message::{{closure}}\n 10: <service::task_service::TaskService as containerd_shim_protos::shim::shim_ttrpc_async::Task>::create::{{closure}}\n 11: <containerd_shim_protos::shim::shim_ttrpc_async::CreateMethod as ttrpc::asynchronous::utils::MethodHandler>::handler::{{closure}}\n 12: <tokio::time::timeout::Timeout<T> as core::future::future::Future>::poll\n 13: ttrpc::asynchronous::server::HandlerContext::handle_msg::{{closure}}\n 14: <core::future::poll_fn::PollFn<F> as core::future::future::Future>::poll\n 15: <ttrpc::asynchronous::server::ServerReader as ttrpc::asynchronous::connection::ReaderDelegate>::handle_msg::{{closure}}::{{closure}}\n 16: tokio::runtime::task::core::Core<T,S>::poll\n 17: tokio::runtime::task::harness::Harness<T,S>::poll\n 18: tokio::runtime::scheduler::multi_thread::worker::Context::run_task\n 19: tokio::runtime::scheduler::multi_thread::worker::Context::run\n 20: tokio::runtime::context::runtime::enter_runtime\n 21: tokio::runtime::scheduler::multi_thread::worker::run\n 22: <tokio::runtime::blocking::task::BlockingTask<T> as core::future::future::Future>::poll\n 23: tokio::runtime::task::core::Core<T,S>::poll\n 24: tokio::runtime::task::harness::Harness<T,S>::poll\n 25: tokio::runtime::blocking::pool::Inner::run\n 26: std::sys::backtrace::__rust_begin_short_backtrace\n 27: core::ops::function::FnOnce::call_once{{vtable.shim}}\n 28: std::sys::pal::unix::thread::Thread::new::thread_start"): unknown
# Exit Code: 128
# Started: Thu, 01 Jan 1970 00:00:00 +0000
# Finished: Thu, 31 Jul 2025 09:51:57 +0000
# Ready: False
# Restart Count: 3
# Environment: <none>
# Mounts:
# /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-pphmm (ro)
# Devices:
# /dev/trusted_store from trusted-storage
# Conditions:
# Type Status
# PodReadyToStartContainers True
# Initialized True
# Ready False
# ContainersReady False
# PodScheduled True
# Volumes:
# trusted-storage:
# Type: PersistentVolumeClaim (a reference to a PersistentVolumeClaim in the same namespace)
# ClaimName: trusted-pvc
# ReadOnly: false
# kube-api-access-pphmm:
# Type: Projected (a volume that contains injected data from multiple sources)
# TokenExpirationSeconds: 3607
# ConfigMapName: kube-root-ca.crt
# ConfigMapOptional: <nil>
# DownwardAPI: true
# QoS Class: BestEffort
# Node-Selectors: katacontainers.io/kata-runtime=true
# Tolerations: node.kubernetes.io/not-ready:NoExecute op=Exists for 300s
# node.kubernetes.io/unreachable:NoExecute op=Exists for 300s
# Events:
# Type Reason Age From Message
# ---- ------ ---- ---- -------
# Normal Scheduled 119s default-scheduler Successfully assigned kata-containers-k8s-tests/large-image-pod to aks-nodepool1-26885032-vmss000000
# Normal SuccessfulMountVolume 119s kubelet MapVolume.MapPodDevice succeeded for volume "trusted-block-pv" globalMapPath "/var/lib/kubelet/plugins/kubernetes.io~local-volume/volumeDevices/trusted-block-pv"
# Normal SuccessfulMountVolume 119s kubelet MapVolume.MapPodDevice succeeded for volume "trusted-block-pv" volumeMapPath "/var/lib/kubelet/pods/b013a29d-76d3-41ee-a827-eb1fb94d068c/volumeDevices/kubernetes.io~local-volume"
# Warning Failed 85s kubelet Error: failed to create containerd task: failed to create shim task: Others("failed to handle message create container\n\nCaused by:\n 0: agent create container\n 1: rpc status: Status { code: DEADLINE_EXCEEDED, message: \"timeout\", details: [], special_fields: SpecialFields { unknown_fields: UnknownFields { fields: None }, cached_size: CachedSize { size: 0 } } }\n\nStack backtrace:\n 0: anyhow::error::<impl core::convert::From<E> for anyhow::Error>::from\n 1: agent::kata::agent::<impl agent::Agent for agent::kata::KataAgent>::create_container::{{closure}}\n 2: virt_container::container_manager::container::Container::create::{{closure}}\n 3: <virt_container::container_manager::manager::VirtContainerManager as common::container_manager::ContainerManager>::create_container::{{closure}}\n 4: runtimes::manager::RuntimeHandlerManager::handler_task_message::{{closure}}::{{closure}}\n 5: runtimes::manager::RuntimeHandlerManager::handler_task_message::{{closure}}\n 6: <service::task_service::TaskService as containerd_shim_protos::shim::shim_ttrpc_async::Task>::create::{{closure}}\n 7: <containerd_shim_protos::shim::shim_ttrpc_async::CreateMethod as ttrpc::asynchronous::utils::MethodHandler>::handler::{{closure}}\n 8: <tokio::time::timeout::Timeout<T> as core::future::future::Future>::poll\n 9: ttrpc::asynchronous::server::HandlerContext::handle_msg::{{closure}}\n 10: <core::future::poll_fn::PollFn<F> as core::future::future::Future>::poll\n 11: <ttrpc::asynchronous::server::ServerReader as ttrpc::asynchronous::connection::ReaderDelegate>::handle_msg::{{closure}}::{{closure}}\n 12: tokio::runtime::task::core::Core<T,S>::poll\n 13: tokio::runtime::task::harness::Harness<T,S>::poll\n 14: tokio::runtime::scheduler::multi_thread::worker::Context::run_task\n 15: tokio::runtime::scheduler::multi_thread::worker::Context::run\n 16: tokio::runtime::context::runtime::enter_runtime\n 17: tokio::runtime::scheduler::multi_thread::worker::run\n 18: <tokio::runtime::blocking::task::BlockingTask<T> as core::future::future::Future>::poll\n 19: tokio::runtime::task::core::Core<T,S>::poll\n 20: tokio::runtime::task::harness::Harness<T,S>::poll\n 21: tokio::runtime::blocking::pool::Inner::run\n 22: std::sys::backtrace::__rust_begin_short_backtrace\n 23: core::ops::function::FnOnce::call_once{{vtable.shim}}\n 24: std::sys::pal::unix::thread::Thread::new::thread_start"): unknown
# Warning Failed 84s kubelet Error: failed to create containerd task: failed to create shim task: Others("failed to handle message create container\n\nCaused by:\n 0: do handle device\n 1: failed to add deivce\n 2: hotplug block device\n 3: device_add Qapi(Error { class: GenericError, desc: \"Failed to get \\\"write\\\" lock\", id: None })\n\nStack backtrace:\n 0: anyhow::error::<impl anyhow::Error>::msg\n 1: hypervisor::qemu::qmp::Qmp::hotplug_block_device\n 2: hypervisor::qemu::inner::QemuInner::hotplug_device\n 3: <hypervisor::qemu::Qemu as hypervisor::Hypervisor>::add_device::{{closure}}\n 4: <hypervisor::device::driver::virtio_blk::BlockDevice as hypervisor::device::Device>::attach::{{closure}}\n 5: hypervisor::device::device_manager::do_handle_device::{{closure}}\n 6: virt_container::container_manager::container::Container::create::{{closure}}\n 7: <virt_container::container_manager::manager::VirtContainerManager as common::container_manager::ContainerManager>::create_container::{{closure}}\n 8: runtimes::manager::RuntimeHandlerManager::handler_task_message::{{closure}}::{{closure}}\n 9: runtimes::manager::RuntimeHandlerManager::handler_task_message::{{closure}}\n 10: <service::task_service::TaskService as containerd_shim_protos::shim::shim_ttrpc_async::Task>::create::{{closure}}\n 11: <containerd_shim_protos::shim::shim_ttrpc_async::CreateMethod as ttrpc::asynchronous::utils::MethodHandler>::handler::{{closure}}\n 12: <tokio::time::timeout::Timeout<T> as core::future::future::Future>::poll\n 13: ttrpc::asynchronous::server::HandlerContext::handle_msg::{{closure}}\n 14: <core::future::poll_fn::PollFn<F> as core::future::future::Future>::poll\n 15: <ttrpc::asynchronous::server::ServerReader as ttrpc::asynchronous::connection::ReaderDelegate>::handle_msg::{{closure}}::{{closure}}\n 16: tokio::runtime::task::core::Core<T,S>::poll\n 17: tokio::runtime::task::harness::Harness<T,S>::poll\n 18: tokio::runtime::scheduler::multi_thread::worker::Context::run_task\n 19: tokio::runtime::scheduler::multi_thread::worker::Context::run\n 20: tokio::runtime::context::runtime::enter_runtime\n 21: tokio::runtime::scheduler::multi_thread::worker::run\n 22: <tokio::runtime::blocking::task::BlockingTask<T> as core::future::future::Future>::poll\n 23: tokio::runtime::task::core::Core<T,S>::poll\n 24: tokio::runtime::task::harness::Harness<T,S>::poll\n 25: tokio::runtime::blocking::pool::Inner::run\n 26: std::sys::backtrace::__rust_begin_short_backtrace\n 27: core::ops::function::FnOnce::call_once{{vtable.shim}}\n 28: std::sys::pal::unix::thread::Thread::new::thread_start"): unknown
# Normal Pulled 41s (x4 over 115s) kubelet Container image "ghcr.io/confidential-containers/test-container:rust-1.79.0" already present on machine
# Normal Created 41s (x4 over 115s) kubelet Created container: app-container
# Warning Failed 41s (x2 over 70s) kubelet Error: failed to create containerd task: failed to create shim task: Others("failed to handle message create container\n\nCaused by:\n 0: do handle device\n 1: failed to add deivce\n 2: hotplug block device\n 3: blockdev-add backend Qapi(Error { class: GenericError, desc: \"Duplicate nodes with node-name='drive-1'\", id: None })\n\nStack backtrace:\n 0: anyhow::error::<impl anyhow::Error>::msg\n 1: hypervisor::qemu::qmp::Qmp::hotplug_block_device\n 2: hypervisor::qemu::inner::QemuInner::hotplug_device\n 3: <hypervisor::qemu::Qemu as hypervisor::Hypervisor>::add_device::{{closure}}\n 4: <hypervisor::device::driver::virtio_blk::BlockDevice as hypervisor::device::Device>::attach::{{closure}}\n 5: hypervisor::device::device_manager::do_handle_device::{{closure}}\n 6: virt_container::container_manager::container::Container::create::{{closure}}\n 7: <virt_container::container_manager::manager::VirtContainerManager as common::container_manager::ContainerManager>::create_container::{{closure}}\n 8: runtimes::manager::RuntimeHandlerManager::handler_task_message::{{closure}}::{{closure}}\n 9: runtimes::manager::RuntimeHandlerManager::handler_task_message::{{closure}}\n 10: <service::task_service::TaskService as containerd_shim_protos::shim::shim_ttrpc_async::Task>::create::{{closure}}\n 11: <containerd_shim_protos::shim::shim_ttrpc_async::CreateMethod as ttrpc::asynchronous::utils::MethodHandler>::handler::{{closure}}\n 12: <tokio::time::timeout::Timeout<T> as core::future::future::Future>::poll\n 13: ttrpc::asynchronous::server::HandlerContext::handle_msg::{{closure}}\n 14: <core::future::poll_fn::PollFn<F> as core::future::future::Future>::poll\n 15: <ttrpc::asynchronous::server::ServerReader as ttrpc::asynchronous::connection::ReaderDelegate>::handle_msg::{{closure}}::{{closure}}\n 16: tokio::runtime::task::core::Core<T,S>::poll\n 17: tokio::runtime::task::harness::Harness<T,S>::poll\n 18: tokio::runtime::scheduler::multi_thread::worker::Context::run_task\n 19: tokio::runtime::scheduler::multi_thread::worker::Context::run\n 20: tokio::runtime::context::runtime::enter_runtime\n 21: tokio::runtime::scheduler::multi_thread::worker::run\n 22: <tokio::runtime::blocking::task::BlockingTask<T> as core::future::future::Future>::poll\n 23: tokio::runtime::task::core::Core<T,S>::poll\n 24: tokio::runtime::task::harness::Harness<T,S>::poll\n 25: tokio::runtime::blocking::pool::Inner::run\n 26: std::sys::backtrace::__rust_begin_short_backtrace\n 27: core::ops::function::FnOnce::call_once{{vtable.shim}}\n 28: std::sys::pal::unix::thread::Thread::new::thread_start"): unknown
# Warning BackOff 4s (x8 over 84s) kubelet Back-off restarting failed container app-container in pod large-image-pod_kata-containers-k8s-tests(b013a29d-76d3-41ee-a827-eb1fb94d068c)
# Name: large-image-pod
# Namespace: kata-containers-k8s-tests
# Priority: 0
# Runtime Class Name: kata
# Service Account: default
# Node: aks-nodepool1-26885032-vmss000000/10.224.0.4
# Start Time: Thu, 31 Jul 2025 09:50:39 +0000
# Labels: <none>
# Annotations: io.containerd.cri.runtime-handler: kata-qemu-runtime-rs-coco-dev
# io.katacontainers.config.agent.policy:
# IyBDb3B5cmlnaHQgKGMpIDIwMjMgTWljcm9zb2Z0IENvcnBvcmF0aW9uCiMKIyBTUERYLUxpY2Vuc2UtSWRlbnRpZmllcjogQXBhY2hlLTIuMAojCgpwYWNrYWdlIGFnZW50X3BvbG...
# io.katacontainers.config.hypervisor.kernel_params: agent.secure_storage_integrity=true
# Status: Running
# IP: 10.244.0.193
# IPs:
# IP: 10.244.0.193
# Containers:
# app-container:
# Container ID: containerd://88c910d6a72a26f8fb7f4fc8c6702ebe1ed3a5b86184bfd5d2c2d6270fa81ea2
# Image: ghcr.io/confidential-containers/test-container:rust-1.79.0
# Image ID: ghcr.io/confidential-containers/test-container@sha256:981b60160a09698712d30c238d870523dc5e94dfaaa0da18d9ba6fb5f4734552
# Port: <none>
# Host Port: <none>
# Command:
# /bin/sh
# -c
# Args:
# sleep 6000
# State: Waiting
# Reason: CrashLoopBackOff
# Last State: Terminated
# Reason: StartError
# Message: failed to create containerd task: failed to create shim task: Others("failed to handle message crSource: kata-containers/kata-containers