TDX: kata-deploy OVMF eagerly accepts all guest memory — large-memory CC pods (≥64GB) hang at boot, exceed 1200s timeout

Summary

The OVMF/TDVF firmware shipped in kata-deploy v4.0.0 (/opt/kata/share/ovmf/OVMF.inteltdx.fd, 4 MB, dated 2025-07-18) performs eager memory acceptance for Intel TDX guests. With large memory VMs (≥ 64 GB), the firmware spends all its time in TDCALL [TDG.MEM.PAGE.ACCEPT] before the guest kernel even starts, causing the kata shim's create_container_timeout (1200s default) to expire and the sandbox to be killed.

This blocks any large-memory TDX confidential workload — e.g. a 2.5 TB LLM inference pod (Kimi-K3, 8× B300 GPU passthrough) cannot boot at all. The OVMF firmware never reaches BdsDxe (no serial output, 0 RSS, 100% CPU in qemu) within 90+ seconds for 64 GB, while the same node's distro ovmf-inteltdx (Ubuntu 2025.11, with lazy-accept) boots the same 64 GB TDX VM in < 15 seconds.

Root Cause

The EDK2 lazy-accept feature (PR #3572, merged 2022) allows OVMF to accept only a small portion of memory (default 512 MB in the IntelTdxX64 Config-B build) and tag the rest as EfiGcdMemoryTypeUnaccepted for the Linux kernel to accept lazily. This is controlled by the PCD PcdLazyAcceptPartialMemorySize (in MB):

• Default 0 in OvmfPkgX64 (Config-A) → lazy accept DISABLED → eager acceptance of ALL memory
• Default 512 in IntelTdxX64 (Config-B) → lazy accept ENABLED → only 512 MB accepted in firmware

The kata-deploy OVMF appears to be a Config-A build with PcdLazyAcceptPartialMemorySize=0 (eager acceptance). The guest kernel has CONFIG_UNACCEPTED_MEMORY=y and supports lazy acceptance, but never gets the chance because the firmware accepts everything first.

TDX does not support memory hotplug (confirmed in kata-containers/src/runtime-rs/crates/hypervisor/src/qemu/cmdline_generator.rs:279-282), so all memory is cold-plugged and must be accepted by OVMF — there is no way to defer it to a hotplug phase.

Evidence

Node: am-b300-60 (Intel TDX, HGX B300 8× GPU)

Test 1 — kata-deploy OVMF, 64 GB TDX VM (no GPU, no kernel, just firmware):

$ timeout 90 /opt/kata/bin/qemu-system-x86_64-tdx-experimental \
    -machine q35,accel=kvm,kernel_irqchip=split,confidential-guest-support=tdx \
    -cpu host,pmu=off -m 65536M -object tdx-guest,id=tdx \
    -nodefaults -no-reboot -serial file:serial.log \
    -bios /opt/kata/share/ovmf/OVMF.inteltdx.fd
# Result: 90+ seconds, 0 bytes serial output, qemu in 100% CPU loop, 0 RSS

Test 2 — host distro ovmf-inteltdx (Ubuntu 2025.11-3ubuntu7), same 64 GB TDX VM:

$ timeout 90 /opt/kata/bin/qemu-system-x86_64-tdx-experimental \
    -machine q35,accel=kvm,kernel_irqchip=split,confidential-guest-support=tdx \
    -cpu host,pmu=off -m 65536M -object tdx-guest,id=tdx \
    -nodefaults -no-reboot -serial file:serial.log \
    -bios /usr/share/ovmf/OVMF.inteltdx.ms.fd
# Result: booted in <15 seconds, 137 bytes serial output:
#   BdsDxe: No bootable option or device was found.
#   BdsDxe: Press any key to enter the Boot Manager Menu.

Test 3 — host distro OVMF, 512 GB TDX VM:

# Same as above but -m 524288M (512 GB)
# Result: booted in <15 seconds, identical serial output

Kata config (runtime-rs)

# /opt/kata/share/defaults/kata-containers/runtime-rs/runtimes/qemu-nvidia-gpu-tdx-runtime-rs/configuration-qemu-nvidia-gpu-tdx-runtime-rs.toml
path = "/opt/kata/bin/qemu-system-x86_64-tdx-experimental"
firmware = "/opt/kata/share/ovmf/OVMF.inteltdx.fd"
confidential_guest = true
default_memory = 8192          # 8 GiB — only used when no pod memory limit
static_sandbox_resource_mgmt = true   # overrides default_memory when pod has memory limit
enable_virtio_mem = false      # no memory hotplug for confidential guests

With a pod memory: 2500Gi request, kata's static_sandbox_resource_mgmt sets default_memory = overhead_memory(512) + 2560000 = 2560512 MiB, so qemu gets -m 2560512M. OVMF tries to eagerly accept all 2.5 TB → hangs for hours → 1200s timeout.

Guest kernel

$ grep -iE "UNACCEPTED|ACCEPT_MEMORY" /opt/kata/share/kata-containers/config-6.18.35-200-nvidia-gpu
CONFIG_UNACCEPTED_MEMORY=y

The kernel supports lazy acceptance — the firmware just never offers it.

Impact

• All large-memory TDX confidential workloads are blocked. Any pod with memory > ~32 GB under kata-qemu-nvidia-gpu-tdx-runtime-rs will fail to boot within the 1200s timeout.
• This affects the entire CC GPU inference use case (LLM serving, where 1-3 TB host RAM is normal for model weights + KV cache headroom).
• The related issue #13180 (create_container_timeout annotation ignored) is a symptom of this — even with a 7200s timeout, eager acceptance of 2.5 TB would take hours.
• AMD SEV-SNP has the same class of problem (#12369) and uses accept_memory=lazy kernel param + the AMDSEV.fd (which has lazy-accept enabled).

Proposed Solutions

Option A (quick win): Ship a Config-B (IntelTdxX64) OVMF with lazy-accept enabled

Replace the kata-deploy OVMF.inteltdx.fd with a build of OvmfPkg/IntelTdx/IntelTdxX64.dsc (Config-B), which has PcdLazyAcceptPartialMemorySize=512 by default. This accepts only 512 MB in firmware and lets the kernel lazily accept the rest.

Build:

build -p OvmfPkg/IntelTdx/IntelTdxX64.dsc -a X64 -t GCC5 -b RELEASE
# or with a custom partial size:
build -p OvmfPkg/IntelTdx/IntelTdxX64.dsc -a X64 -t GCC5 -b RELEASE -D LAZY_ACCEPT_PARTIAL_MEM=1024

Option B: Add accept_memory=lazy to the default kernel_params

The kata TDX config already has kernel_params = "cgroup_no_v1=all pci=realloc ...". Adding accept_memory=lazy would tell the kernel to accept memory lazily — but this only works if the OVMF provides the unaccepted memory table (i.e. the firmware must still have lazy-accept enabled). So this is a complement to Option A, not a replacement.

Option C: Document the workaround

Until a Config-B OVMF ships, document that users can replace /opt/kata/share/ovmf/OVMF.inteltdx.fd with their distro's ovmf-inteltdx package (e.g. Ubuntu's /usr/share/ovmf/OVMF.inteltdx.ms.fd, which has lazy-accept) — verified to boot a 512 GB TDX VM in <15s on the same node where the kata OVMF hangs on 64 GB.

Environment

Related

• #13180 — create_container_timeout annotation ignored (symptom of this on large-memory pods)
• #12369 — SEV-SNP default_memory over 2GB hang (same class, AMD side)
• edk2 PR #3572 — Introduce Lazy-accept for Tdx guest (merged 2022)
• edk2 issue #10306 — Customize lazy-accepted memory size in build time for OVMF
• edk2 issue #9262 — Enable Multi-core based lazy-accept for TDVF