[Bug]: HTTPChaos Experiment Remains in Injecting State on Subsequent Injection
Chaos Mesh Version
v2.8.2
Kubernetes Version
v1.35.4
Describe the bug
When applying an HTTP Chaos experiment for the first time, the experiment behaves as expected: the target pods are successfully injected, and once the experiment completes, all affected pods return to their normal state(AllRecovered: True).
However, when applying a similar HTTP Chaos experiment targeting the same pods after the first experiment has completed, the behavior becomes inconsistent.
In some cases, the second experiment fails to complete the injection process and remains indefinitely in the Injecting state.
HTTPChaos Manifest
kind: HTTPChaos
apiVersion: chaos-mesh.org/v1alpha1
metadata:
namespace: sit08
name: manual-2-delphi-game-management-1785240061563
spec:
selector:
namespaces:
- sit08
labelSelectors:
app.kubernetes.io/instance: delphi-game-management
mode: all
target: Request
abort: true
port: 7040
path: /delphi-game-management/api/v1/games/aee2eecb-9b21-4fde-87d1-0284cfacf018/activate
method: PUT
duration: 15sDeployemnt steps
- Deploy the target application.
- Apply the above HTTPChaos experiment.
- Verify that the experiment is successfully injected.
- Wait for the experiment to complete and confirm that the pods have fully recovered.
- Apply another HTTPChaos experiment with the same configuration (or an equivalent one) targeting the same pods.
- Observe the experiment state.
Expected Behavior The second experiment should behave identically to the first:
- Successfully inject into the target pods.
- Complete after the configured duration.
Controller Logs
failed to apply for pod sit08/delphi-game-management-7d5f95ff9-qxdrs:
rpc error: code = Unknown desc = create http chaos:
get PID of container(containerd://ab2f9fddebb1fa95c5a113db534ac753f901ce82fdd52f02e7148ba55dd58ba2):
no running task found:
task ab2f9fddebb1fa95c5a113db534ac753f901ce82fdd52f02e7148ba55dd58ba2 not found: not found
{
"type": "Warning",
"object": {
"kind": "PodHttpChaos",
"namespace": "sit08",
"name": "delphi-game-management-7d5f95ff9-qxdrs",
"uid": "a8a9d7a5-9b12-4e0c-affc-f2f71c7b899c",
"apiVersion": "chaos-mesh.org/v1alpha1",
"resourceVersion": "14759412"
},
"reason": "Failed"
}Chaos Daemon Logs
2:01:47.088054Z ERROR handle{proxy=Proxy { opt: ProxyOpt { ipc_path: "/tmp/58c177b1-f93c-4733-b4a7-5273da237bff.sock", verbose: 2 }, net_env: NetEnv { netns: "fda298e9-5aa1ns", device: "eth0", ip: "10.131.15.70/32", bridge1: "fda298e9-5aa1b1", bridge2: "fda298e9-5aa1b2", veth1: "fda298e9-5aa1v1", veth2: "veth0", veth3: "veth1", veth4: "fda298e9-5aa1v4", save_routes: [RouteMessage { header: RouteHeader { address_family: 2, destination_prefix_length: 32, source_prefix_length: 0, tos: 0, table: 254, protocol: 3, scope: 253, kind: 1, flags: (empty) }, nlas: [Table(254), Destination([10, 131, 15, 108]), Oif(703)] }, RouteMessage { header: RouteHeader { address_family: 2, destination_prefix_length: 0, source_prefix_length: 0, tos: 0, table: 254, protocol: 3, scope: 0, kind: 1, flags: (empty) }, nlas: [Table(254), Metrics([8, 0, 2, 0, 247, 34, 0, 0]), Gateway([10, 131, 15, 108]), Oif(703)] }] }, rtnl_handle: Handle(ConnectionHandle { requests_tx: UnboundedSender(Some(UnboundedSenderInner { inner: UnboundedInner { state: 9223372036854775808, message_queue: Queue { head: 0x7fdf7001d9c0, tail: UnsafeCell { .. } }, num_senders: 1, recv_task: AtomicWaker } })) }), sender: Some(Sender { inner: Some(Inner { state: State { is_complete: false, is_closed: false, is_rx_task_set: true, is_tx_task_set: false } }) }), rx: None, task: Some(JoinHandle { id: Id(41) }) } request=Request { method: PUT, uri: /, version: HTTP/1.1, headers: {"host": "", "user-agent": "Go-http-client/1.1", "content-length": "12"}, body: Body(Streaming) }}: chaos_tproxy::proxy::net::bridge: clear routes del_routes_noblock with error Received a netlink error message No such process (os error 3)
{"namespacedName": "sit08/delphi-game-management-7d5f95ff9-5h2vm"}
ERROR handle{proxy=Proxy { opt: ProxyOpt { ipc_path: "/tmp/58c177b1-f93c-4733-b4a7-5273da237bff.sock", verbose: 2 }, net_env: NetEnv { netns: "fda298e9-5aa1ns", device: "eth0", ip: "10.131.15.70/32", bridge1: "fda298e9-5aa1b1", bridge2: "fda298e9-5aa1b2", veth1: "fda298e9-5aa1v1", veth2: "veth0", veth3: "veth1", veth4: "fda298e9-5aa1v4", save_routes: [RouteMessage { header: RouteHeader { address_family: 2, destination_prefix_length: 32, source_prefix_length: 0, tos: 0, table: 254, protocol: 3, scope: 253, kind: 1, flags: (empty) }, nlas: [Table(254), Destination([10, 131, 15, 108]), Oif(703)] }, RouteMessage { header: RouteHeader { address_family: 2, destination_prefix_length: 0, source_prefix_length: 0, tos: 0, table: 254, protocol: 3, scope: 0, kind: 1, flags: (empty) }, nlas: [Table(254), Metrics([8, 0, 2, 0, 247, 34, 0, 0]), Gateway([10, 131, 15, 108]), Oif(703)] }] }, rtnl_handle: Handle(ConnectionHandle { requests_tx: UnboundedSender(Some(UnboundedSenderInner { inner: UnboundedInner { state: 92233720368Any idea on what could be the case for this issue ?
Source: chaos-mesh/chaos-mesh