agent: env keys and values are trimmed after policy evaluation, so the process does not get the approved bytes

Summary

The agent trims leading and trailing whitespace (Rust str::trim) from every environment variable key and value just before it execvps a container process, for container start and for exec. The agent policy has already been evaluated on the untrimmed request, and genpolicy's allow_var compares entries byte for byte. A policy that lists a value with leading or trailing whitespace therefore approves bytes that the process never receives.

For confidential containers the policy is part of the measured init-data (bound to HOST_DATA on SEV-SNP, MRCONFIGID on TDX). The measurement then does not describe the process environment byte for byte. runc delivers the same values unchanged. The OCI runtime spec gives process.env the semantics of POSIX environ, which does not trim.

We reproduced it with a genpolicy-generated policy on SEV-SNP: the Pod starts, the process gets the trimmed values, and the same Pod with the trimmed value in its spec is refused by that policy.

Environment

• Kata Containers 4.0.0 (cf82bb35c803): Go shim and agent (confidential guest image) from kata-deploy 4.0.0 (quay.io/kata-containers/kata-deploy:4.0.0); genpolicy from kata-tools-static-4.0.0-amd64.tar.zst (sha256 4fa1cabd8d6e757c271caa991b889e1c2ccdbbdce58872b8b581720b3867be36)
• Runtime class kata-qemu-snp from kata-deploy 4.0.0, installed with a non-default install prefix; kernel, image and firmware are the release's SNP defaults (QEMU 11.0.1 from the release with confidential-guest-support=snp, containerd snapshotter nydus-for-kata-tee), guest kernel 6.18.35
• Kubernetes v1.36.3 (k0s), containerd 2.3.3, host Ubuntu 26.04 with kernel 7.0.0-22-generic, AMD EPYC 9454 with SEV-SNP
• Image python:3.13-alpine (sha256:79e7a9b9ff1cbceff819f856fb374477792a5967759d94df266de7b7b4120e6f at the time of the runs)
• Comparison: the same Pods under runc on the same node

Where it happens

At 4.0.0:

• src/libs/kata-sys-util/src/validate.rs:44: valid_env() checks for NUL, = in the key and an empty key, then returns Some((key.trim(), value.trim())).
• src/agent/rustjail/src/container.rs:745-756: the forked child clears its environment, calls valid_env() on each OCI process.env entry and env::set_var(key, value), then execs. Container start and exec share this path.
• src/agent/src/rpc.rs:924 (create_container) and :957 (exec_process): is_allowed(&req) evaluates the policy on the untrimmed request before do_create_container / do_exec_process.
• src/tools/genpolicy/rules.rego:953-957: allow_var rule 1 is p_var == i_var (rule 8 is the same comparison after substituting $(sandbox-namespace)). genpolicy renders each entry as format!("{}={value}", name) without trimming (src/tools/genpolicy/src/pod.rs:698-721), so a generated policy holds the untrimmed value.
• OCI hook environments go through the same function (src/libs/kata-sys-util/src/hooks.rs:243).

test_valid_env asserts the trim ("a= b" and "a=b " both give ("a", "b")), so it is the function's intended behaviour. It came in with 57bb7ffae (2021, the fix for #3006, where an empty key made env::set_var panic) and moved to kata-sys-util in 85f9094f1 (#5858). Before that the child called env::set_var(v[0], v[1]) untrimmed. The panic guard does not need the trim: set_var accepts whitespace in keys and values.

Observed

All outputs in this issue were observed on 2026-09-25 on the environment above. The commands are under "Minimal reproduction".

The probe Pod from "Minimal reproduction", with the ten values below in place of its six, ran under runc and under kata-qemu-snp on the same node. It has no init-data annotation, so the guest used its built-in policy, and this run shows the trim alone. kubectl exec of the same print loop into each running container printed the same bytes as the container process.

Only whitespace at the ends is removed, including Unicode White_Space such as U+00A0 and U+2003. Inner whitespace, an inner \r\n and = are kept.

The agent receives the values untrimmed:

• On the host, the OCI bundle config.json that containerd wrote for the Kata container holds every value untrimmed.
• Under a generated policy the Pod starts and the process gets the trimmed values. That policy lists only the untrimmed values, and for these entries only allow_var rules 1 and 8 can match, both by byte equality. So the CreateContainerRequest the agent admitted carried the untrimmed bytes.
• The agent's policy trace prints the request it refused. The ExecProcessRequest for the running container held \"T_TRAILING_NL=value\\n\", \"T_LEADING_WS= \\t lead\" and \"T_UNICODE_WS=\\\u{a0}nbsp\\\u{2003}\". In the CreateContainerRequest refused after only T_TRAILING_NL was changed to value, the other values were untrimmed too.

The shim does not trim either (from the code): for container create it copies the OCI spec into the agent request field by field (grpc.OCItoGRPC, called from src/runtime/virtcontainers/kata_agent.go:1683), and for exec it splits each entry at the first = and joins it again (getEnvs in src/runtime/pkg/containerd-shim-v2/exec.go:53-70, then cmdEnvsToStringSlice in kata_agent.go:583-592).

We found this on another workload, whose generated policy listed a PEM public key with its final newline. The policy admitted the container. The process got the key without the newline, and a parser that requires the final newline refused it at startup.

Impact

• This does not look like a policy bypass. The host can only send entries the policy admits, and the trim is a deterministic function of them. The default allow_env_regex patterns are anchored and admit no whitespace.
• The attested init-data does not describe the process environment byte for byte. A relying party that takes expected values from the attested policy gets the wrong bytes for any value with edge whitespace. A measured non-empty value can arrive empty (T_ONLY_WS below). Two policies that differ only in such whitespace give different HOST_DATA values but start processes with the same environment.
• Workloads that work under runc can fail under Kata. Examples are PEM blocks, values that need a trailing newline, and values that are whitespace on purpose. The failure shows up in the application, and nothing in the policy or the agent logs points at the environment.

Status on main

Unchanged as of main 8a2c6acfa662 (2026-09-25). validate.rs is the same blob (04a1808e340a) at 4.0.0, 4.1.0 (ddcb1ad8d23c), 4.2.0 (c7351e797eff) and main. The call site is now container.rs:765-771. On main, create_container and exec_process still call is_allowed(&req) before doing the work (rpc.rs:956 and :989). The allow_env / allow_var rules in rules.rego are identical to 4.0.0. Searching issues and PRs in this repository for valid_env, trim, whitespace and allow_var found nothing about this.

Related observation (from reading the code, not run end to end)

Before #5858, valid_env() checked that the key was non-empty after trimming it. Since #5858 it checks the raw key and trims afterwards. A key made only of whitespace therefore passes the check and reaches env::set_var("", value), which panics. That is the #3006 failure again.

Kubernetes admits such names. With RelaxedEnvironmentVariableValidation (beta and on by default since 1.32, GA and locked on since 1.34), Pod validation checks env names with IsRelaxedEnvVarName (staging/src/k8s.io/apimachinery/pkg/util/validation/validation.go:388-403 at v1.36.3), which accepts any printable ASCII character except =. Go's unicode.IsPrint counts the ASCII space as printable, so names such as " " and "A " pass API validation; a server-side dry run (kubectl apply --dry-run=server) of a Pod with both names was accepted by the v1.36.3 API server above. The second is the key-side form of the trim above: a policy that lists A =x approves an entry the process receives as A=x. We have not run either case in a Kata guest. Option A in full (no key trim) removes both.

Options

A. Stop trimming (suggested). valid_env() keeps its NUL, = and empty-key checks and returns the key and value unchanged. Update test_valid_env to match. The process then gets exactly the bytes the policy approved, as it would under runc. The risk is workloads that depend on the trim without knowing it. A release note covers that. If keys are to keep some normalisation, stopping the value trim alone fixes the measurement mismatch for values. The empty-key check must then run on the trimmed key, as it did before #5858, or the whitespace-only key still reaches env::set_var("").

B. Keep the trim and reject what it would change. genpolicy refuses (or at least warns about) env keys and values with leading or trailing whitespace. rules.rego refuses an input entry whose key or value has edge whitespace. The whitespace set must match Rust's char::is_whitespace, which is what str::trim uses. Every entry the policy admits then arrives unchanged. The cost: Pods with such values cannot run under a generated policy. They do not get their values today either, but they fail without any error.

C. Document it. State in the agent and policy docs that env keys and values are trimmed (Unicode White_Space, both ends) after policy evaluation and before exec. Policy authors and verifiers can then allow for it. The mismatch stays.

A plus a release note is the smallest change. C could go in the docs either way.

Minimal reproduction

envtrim-kata.yaml:

apiVersion: v1
kind: Pod
metadata:
  name: envtrim-kata
spec:
  runtimeClassName: kata-qemu-snp
  restartPolicy: Never
  securityContext:
    supplementalGroups: [1, 2, 3, 4, 6, 10, 11, 20, 26, 27]   # see "With a generated policy"
  containers:
  - name: probe
    image: python:3.13-alpine
    command:
    - python3
    - -c
    - |
      import os, time
      for k in sorted(os.environb):
          if k.startswith(b"T_"):
              v = os.environb[k]
              print(k.decode(), len(v), v.hex(), repr(v), flush=True)
      time.sleep(600)
    env:
    - {name: T_TRAILING_NL,     value: "value\n"}
    - {name: T_TRAILING_SPACES, value: "value   "}
    - {name: T_LEADING_WS,      value: " \t lead"}
    - {name: T_ONLY_WS,         value: " \n "}
    - {name: T_UNICODE_WS,      value: "\u00a0nbsp\u2003"}
    - {name: T_INNER,           value: "a \n b"}

T_UNICODE_WS uses YAML escapes for U+00A0 (no-break space) and U+2003 (em space), so copying the file cannot turn them into plain spaces. supplementalGroups is needed only for the generated-policy run below (see there); it does not change the environment.

envtrim-runc.yaml is the same file with the runtimeClassName line removed and name: envtrim-runc.

kubectl apply -f envtrim-kata.yaml -f envtrim-runc.yaml
kubectl wait --for=condition=Ready pod/envtrim-kata pod/envtrim-runc --timeout=300s
kubectl logs envtrim-runc
kubectl logs envtrim-kata
kubectl exec envtrim-kata -- python3 -c 'import os; print(repr(os.environb[b"T_TRAILING_NL"]))'
kubectl exec envtrim-runc -- python3 -c 'import os; print(repr(os.environb[b"T_TRAILING_NL"]))'
kubectl delete pod envtrim-kata envtrim-runc

We ran these with -n pointing at a scratch namespace. Both Pods were Ready about 10 seconds after kubectl apply. Without an init-data annotation the guest uses its built-in policy (src/kata-opa/allow-all.rego unless the guest image was built with another one, see tools/osbuilder/rootfs-builder/rootfs.sh), which allowed kubectl logs and kubectl exec. We ran kata-qemu-snp only. The trim is in the agent, so other runtime classes that use this agent should behave the same, but we have not run them.

runc (the exec printed b'value\n'):

T_INNER 5 61200a2062 b'a \n b'
T_LEADING_WS 7 2009206c656164 b' \t lead'
T_ONLY_WS 3 200a20 b' \n '
T_TRAILING_NL 6 76616c75650a b'value\n'
T_TRAILING_SPACES 8 76616c7565202020 b'value   '
T_UNICODE_WS 9 c2a06e627370e28083 b'\xc2\xa0nbsp\xe2\x80\x83'

Kata 4.0.0 (the exec printed b'value'):

T_INNER 5 61200a2062 b'a \n b'
T_LEADING_WS 4 6c656164 b'lead'
T_ONLY_WS 0  b''
T_TRAILING_NL 5 76616c7565 b'value'
T_TRAILING_SPACES 5 76616c7565 b'value'
T_UNICODE_WS 4 6e627370 b'nbsp'

With a generated policy

We used genpolicy, rules.rego and genpolicy-settings.json from kata-tools-static-4.0.0-amd64.tar.zst (opt/kata/bin/genpolicy and opt/kata/share/defaults/kata-containers/) unchanged, plus three drop-ins (see src/tools/genpolicy/drop-in-examples/README.md):

rules.rego
my-settings/
  genpolicy-settings.json
  genpolicy-settings.d/
    20-oci-1.3.0-drop-in.json
    30-read-stream.json
    40-bundle-path.json

• 20-oci-1.3.0-drop-in.json, copied from drop-in-examples/:

  json复制

  [
    {
      "op": "replace",
      "path": "/kata_config/oci_version",
      "value": "1.3.0"
    }
  ]

  containerd 2.3.3 wrote ociVersion 1.3.0 into the bundle. The default is 1.1.0, and CreateContainerRequest requires p_oci.Version == i_oci.Version (rules.rego:106-107).

• 30-read-stream.json:

  json复制

  [{"op":"replace","path":"/request_defaults/ReadStreamRequest","value":true}]

  The default policy denies ReadStreamRequest, and the agent then clears the output it returns (read_stdout / read_stderr in rpc.rs), so kubectl logs would be empty.

• 40-bundle-path.json, needed only because containerd's state directory on k0s is /run/k0s/containerd rather than /run/containerd:

  json复制

  [
    {"op":"replace","path":"/pause_container/Annotations/io.katacontainers.pkg.oci.bundle_path","value":"/run/k0s/containerd/io.containerd.runtime.v2.task/k8s.io/$(bundle-id)"},
    {"op":"replace","path":"/other_container/Annotations/io.katacontainers.pkg.oci.bundle_path","value":"/run/k0s/containerd/io.containerd.runtime.v2.task/k8s.io/$(bundle-id)"}
  ]

genpolicy -p rules.rego -j my-settings/ -y envtrim-kata.yaml
kubectl apply -f envtrim-kata.yaml
kubectl wait --for=condition=Ready pod/envtrim-kata --timeout=300s
kubectl logs envtrim-kata
kubectl exec envtrim-kata -- python3 -c 'import os; print(repr(os.environb[b"T_TRAILING_NL"]))'

genpolicy added the io.katacontainers.config.hypervisor.cc_init_data annotation (base64 of gzipped TOML). Its policy.rego lists "T_TRAILING_NL=value\n", (JSON escape for the newline) in the container Env, and the other five values untrimmed as well. QEMU started the guest with host-data equal to the SHA-256 of that TOML. The Pod became Ready, and kubectl logs envtrim-kata printed exactly the Kata output above.

The kubectl exec was refused, as expected with the default ExecProcessRequest settings (allowed_commands and regex empty). The refusal prints the request the agent received (excerpt, ... marks omitted text):

error: Internal error occurred: Internal error occurred: error executing command in container: failed to exec in container: ... desc = "ExecProcessRequest is blocked by policy: agent_policy:1648: ExecProcessRequest 1: input = { ... \"Env\": [\"PATH=/usr/local/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin\", \"HOSTNAME=envtrim-kata\", \"GPG_KEY=7169605F62C751356D054A26A821E680E5FA6305\", \"PYTHON_VERSION=3.13.15\", \"PYTHON_SHA256=1e66a7945a48390ee4c2a4268a0e4185884059a13c4aab6d148aa208deea4a76\", \"T_TRAILING_NL=value\\n\", \"T_TRAILING_SPACES=value   \", \"T_LEADING_WS= \\t lead\", \"T_ONLY_WS= \\n \", \"T_UNICODE_WS=\\\u{a0}nbsp\\\u{2003}\", \"T_INNER=a \\n b\", \"KUBERNETES_PORT_443_TCP=tcp://10.96.0.1:443\", \"KUBERNETES_PORT_443_TCP_PROTO=tcp\", \"KUBERNETES_PORT_443_TCP_PORT=443\", \"KUBERNETES_PORT_443_TCP_ADDR=10.96.0.1\", \"KUBERNETES_SERVICE_HOST=10.96.0.1\", \"KUBERNETES_SERVICE_PORT=443\", \"KUBERNETES_SERVICE_PORT_HTTPS=443\", \"KUBERNETES_PORT=tcp://10.96.0.1:443\"], ...

Notes on genpolicy 4.0.0 with this Pod:

• Without supplementalGroups it stops with ERROR: guest_pull is enabled for container 'probe' using image 'python:3.13-alpine'. The generated policy expects UID=0, GID=0, AdditionalGids={0, 1, 2, 3, 4, 6, 10, 11, 20, 26, 27}; ... and suggests the list used above.
• It rejects enableServiceLinks (Unsupported field: spec.enableServiceLinks), so the Pod does not set it. The probe prints only T_ variables.
• Without metadata.namespace the generated policy does not pin a namespace (io.kubernetes.cri.sandbox-namespace is empty, and the agent's trace shows allow_namespace 2: no namespace found on policy data), so the Pod can be applied in any namespace.
• genpolicy rewrites the YAML and writes U+00A0 and U+2003 as literal characters.

Then change only the Pod's value to the trimmed bytes and keep the generated annotation. In genpolicy's output the line is value: "value\n", and the sed below matches only that line.

kubectl delete pod envtrim-kata
sed -i 's/value: "value\\n"/value: "value"/' envtrim-kata.yaml
kubectl apply -f envtrim-kata.yaml
kubectl get pod envtrim-kata
kubectl get events --field-selector involvedObject.name=envtrim-kata,reason=Failed -o json

NAME           READY   STATUS       RESTARTS   AGE
envtrim-kata   0/1     StartError   0          14s

The message of the Failed event is 146044 bytes. Excerpt, with ... marking omitted text and a line break added before each agent_policy: of the last part (the omitted allow_var 3 lines are the 15 default allow_env_regex patterns):

Error: failed to create containerd task: failed to create shim task: "CreateContainerRequest is blocked by policy: agent_policy:135: allow_create_container_input: input = { ... \"Env\": [\"PATH=/usr/local/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin\", \"HOSTNAME=envtrim-kata\", \"GPG_KEY=7169605F62C751356D054A26A821E680E5FA6305\", \"PYTHON_VERSION=3.13.15\", \"PYTHON_SHA256=1e66a7945a48390ee4c2a4268a0e4185884059a13c4aab6d148aa208deea4a76\", \"T_INNER=a \\n b\", \"T_TRAILING_NL=value\", \"T_TRAILING_SPACES=value   \", \"T_LEADING_WS= \\t lead\", \"T_ONLY_WS= \\n \", \"T_UNICODE_WS=\\\u{a0}nbsp\\\u{2003}\", \"KUBERNETES_PORT_443_TCP_PROTO=tcp\", \"KUBERNETES_PORT_443_TCP_PORT=443\", \"KUBERNETES_PORT_443_TCP_ADDR=10.96.0.1\", \"KUBERNETES_SERVICE_HOST=10.96.0.1\", \"KUBERNETES_SE