[Bug]: PhysicalMachineChaos: automatic recovery fails with empty uid (404), attack left running indefinitely
Chaos Mesh Version
v2.8.3
Kubernetes Version
v1.33.10
Describe the bug
When a PhysicalMachineChaos experiment's duration expires, the controller-manager attempts automatic recovery by sending a DELETE request to chaosd, but the uid used in that request is empty, resulting in a 404 and the attack being left running on the target indefinitely.
At creation time, chaosd returns a valid uid in the 200 OK response body (e.g. {"status":200,"message":"attack successfully","uid":""}). However, this uid does not appear to be persisted anywhere the controller's Recover() call can read it back from — the generic Record.id field used internally by the reconciler is set to the target address (https://:port), not to the chaosd-generated uid.
Confirmed reproducible with two different actions (stress-mem and disk-fill), so this is not specific to a single action type. Also confirmed independent of leader-election / multiple chaos-controller-manager replicas (identical behavior with a single replica).
Manually calling DELETE /api/attack/{uid} directly against chaosd, using the uid obtained via GET /api/experiments/, works correctly and clears the attack (memory freed / file removed, status moves to destroyed) — confirming the correct recovery endpoint and that chaosd itself behaves correctly when given the right uid.
We also tried setting spec.uid explicitly in the PhysicalMachineChaos YAML (as seen used for network-loss in #4622), hoping the controller would forward it into the create payload and reuse it for recovery. This had no effect — the stress/disk create payload sent to chaosd never includes a uid field regardless of what spec.uid is set to.
To Reproduce
- Deploy chaosd in service mode on a target VM, reachable via mTLS from the cluster.
- Apply a PhysicalMachineChaos experiment with a short duration, e.g.:
kind: PhysicalMachineChaos apiVersion: chaos-mesh.org/v1alpha1 metadata: namespace: name: test-vm-mem-repro spec: action: stress-mem address: - https://:31768 selector: {} mode: all stress-mem: size: 250MB duration: 2m
- Observe controller-manager logs: the create call succeeds and logs a uid.
- Wait for duration to expire.
- Observe controller-manager logs: recover physical machine chaos is logged, followed by experiment not found {"uid": ""} and a 404 from chaosd at DELETE /api/attack/.
- Confirm the attack is still active on the target (e.g. memory usage stays elevated / disk fill file still present).
- Manually run DELETE /api/attack/{uid} against chaosd (uid obtained via GET /api/experiments/) — this succeeds and correctly stops the attack, confirming chaosd's API works as expected when given the right uid.
Reproduced identically with action: disk-fill instead of stress-mem, and with chaos-controller-manager scaled to a single replica.
Source: chaos-mesh/chaos-mesh