Packet handling for non-existing pods with eBPF

When sending packets (e.g. doing a curl request) to a non-existing pod IP, the Calico eBPF behaviour differs depending on where that destination pod was/would be (i.e. in which hosts subnet it was.

Sending traffic to an IP address which ends on the local node, curl fails like the following

*   Trying 100.66.39.2:80...
* Immediate connect fail for 100.66.39.2: Invalid argument
* Failed to connect to 100.66.39.2 port 80 after 0 ms: Could not connect to server
* closing connection #0

Sending traffic to a not-existing IP on a remote node ends in a timeout. This likely is caused by the blackhole route when no more-specific entry is available.

Problem statement:

The dropping of such invalid connects (invalid, because the destination does not exist (yet/anymore)) leads to a problem. Assume an ingress controller, which consumes endpointslices and proxies traffic to pods directly is slow in processing a pod removal. Then it would send traffic to the former pod IP which ends in timeouts. Same issue possibly happens in Calico itself. We currently assume that stale conntrack entries forward traffic to not existing pod IPs - and just cause timeouts. If the connection would be reset properly instead of stealthily dropping the packet, the conntrack entry could be properly marked invalid. This would allow recovering from such a situation faster.

Expected Behavior

I'd expect that initiators of impossible connections or communication partners of former working connections terminate quickly once a connection is dead instead of relying on pure timeouts.

Current Behavior

Connections to not existing IPs on remote nodes end in timeouts.

Possible Solution

What if calico would not create a blackhole route for the local subnet, but a "unreachable" or "prohibit" route? Or if that would be at least configurable - so the admin can decide which mode to use doing a trade-off decision.

Is there any relevant negative consequence of not dropping silently but sending negative replies?

Steps to Reproduce (for bugs)

1. Assume/create 2 nodes with e.g. pod cidrs 100.65.1.0/24 and 100.65.2.0/24
2. node 1 will have local pods from 100.65.1.0/24 and there will be a blackhole route to 100.65.1.0/24 and a vxlan route for 100.65.2.0/24. Inversely for node 2
3. curl -v 100.65.1.1 will cause "could not connect", curl -v 100.65.2.1 will end in timeout

Context

We use the ebpf mode in large setups with many new connections and large conntrack tables (up to 20M entries). Esp. the new sweep mechanism to mark conntrack entries with send-rst is too slow and too many packets go into timeout because no reset is sent. If the destination would properly cause a connection reset, working new connections could be established. This especially is a problem for traffic from outside k8s clusters when a deployment of an ingress controller is redeployed and all pods are recreated.

We suspect an increase of that problem with reduced amounts if source IPs and source ports, e.g. when load balancers with SNAT are in front - stale conntrack entries seem to suffer from collisions/exhaustion of sourceIP/sourePort space which keep stale conntrack entries even longer alive - but we'll likely report this as a seperate issue.

Your Environment

• Calico version: 3.32.1
• Calico dataplane (bpf, nftables, iptables, windows etc.): bpf
• Orchestrator version (e.g. kubernetes, openshift, etc.): kubernetes 1.34/1.35
• Operating System and version: FlatCar ContainerLinux