Intercept multiple services (headless + ClusterIP) for a single StatefulSet
Please describe your use case / problem.
We're migrating an internal CLI tool from mirrord to Telepresence for local development against a StatefulSet ("store") that is fronted by three Kubernetes Services sharing the same named container port (http, containerPort 8080):
store-query: a normal Service with a realClusterIP,targetPort: httpstore-session: headless (clusterIP: None),targetPort: httpstore-gossip: headless (clusterIP: None),targetPort: http
Different real clients in our cluster reach the workload through different services depending on role (query traffic via store-query, application/session traffic via store-session, inter-replica gossip via store-gossip). For a local developer to get a faithful picture of the workload's behavior, traffic arriving via all three services needs to reach their locally-running replacement process, the same way it did under mirrord's steal mode (which intercepts at the process/socket level regardless of which Service/DNS name a client used to reach the pod).
Describe the solution you'd like
A supported way to have a single telepresence intercept redirect traffic to a workload regardless of which of several same-port Services (mixing headless and ClusterIP-backed) was used to reach it, ideally without having to change the Services' targetPort from a name to a number, and without that mechanism swallowing the pod's own kubelet liveness/readiness probe traffic (see below).
Describe alternatives you've considered
telepresence intercept <workload> --service <name> --port <local>:<remote>, trying each of the three service names in turn (--service store-query,--service store-session,--service store-gossip). Result: regardless of which--servicewas specified, only traffic throughstore-query(the one with a realClusterIP) was ever redirected to the local process. Traffic sent directly to the pod IP (which is what headless-service DNS resolves to forstore-session/store-gossip) always reached the real in-cluster container, never the local one. NoinitContainerwas ever injected during this phase, despite the traffic-manager already havingAGENT_INIT_CONTAINER_ENABLED=true.- Passing the headless service's own name directly as the intercept's positional
<name>argument (telepresence intercept store-session --port <local>:<remote>), without--workload. This failed outright:connector.CreateIntercept: workload "store-session" not found, the CLI only resolves<name>/--workloadagainst workload kinds (Deployment/ReplicaSet/StatefulSet/Rollout), never against a Service name, even though the headless-services doc showstelepresence intercept my-headless --port 8080as if the service name were directly usable. - Per the same doc's caveat about
NET_ADMINand "numeric ports," we changedstore-session'stargetPortfrom the namehttpto the number8080and retried the intercept. This did trigger the documented mechanism: atel-agent-initinitContainer appeared for the first time but produced inconsistent and unsafe results rather than a fix:- Traffic through
store-gossip(still namedtargetPort, untouched) started reaching the local process. - Traffic through
store-session(the one we actually pointed at, now numeric) andstore-query(previously working fine) both failed to connect at all (curlexit/HTTP code000). - The pod flipped to
1/2 Ready. Its own kubelet liveness/readiness probes (http-get http://:tm-http/health/live,.../health/ready) started failing, because they also connect directly to the pod IP on the original port, the same traffic class theNET_ADMINiptables rule intercepts. In other words, enabling this mechanism doesn't just redirect Service-routed traffic to the local process; it appears to redirect all inbound traffic to the container's original port, including the kubelet's own probes, with no way we could find to exclude probe traffic. That's a real operational risk (Kubernetes can restart/evict a container whose probes start failing).
- Traffic through
We couldn't find any flag or documented option to (a) scope interception to a set of services rather than exactly one, or (b) enable the numeric-port/NET_ADMIN mechanism without it also capturing the pod's own probe traffic.
Versions (please complete the following information)
OSS Client : v2.31.2
OSS Root Daemon : v2.31.2
OSS User Daemon : v2.31.2
OSS Traffic Manager: v2.31.2
Traffic Agent : ghcr.io/telepresenceio/tel2:2.31.2- Kubernetes:
v1.35.5+k3s1(k3d v5.9.0 / k3s v1.35.5-k3s1), single-node local dev cluster
Additional context
- Container port definition:
{"containerPort": 8080, "name": "http", "protocol": "TCP"}a single named port, referenced by name (targetPort: http) from all three Services. - Happy to provide a minimal StatefulSet + 3-Service YAML repro (one ClusterIP, two headless, all sharing the named port) if useful, the behavior above was fully reproducible across multiple clean pod recreations, not a one-off flake.
Source: telepresenceio/telepresence