backend: multiplexer: connection key ignores the query, so closing one watch kills another on the same path

Describe the bug

I turned the websocket multiplexer on to try it out and the pods list stopped updating after the first event. No error, nothing in the console, socket still open. The pod I deleted just sat there at "Terminating" until I left the page.

The frontend and the backend don't agree on what makes a watch unique. The frontend keys a subscription by cluster + path + query. The backend drops the query (createConnectionKey, multiplexer.go:1133):

return clusterID + ":" + path + ":" + userID

The query is where the resourceVersion lives, so on every event the frontend resubscribes with a new query and closes the old one, and the backend looks up both under the same key. One kubectl delete pod on /wsMultiplexer:

2.0s ->  REQUEST  /api/v1/namespaces/demo/pods  q="watch=1&resourceVersion=8199"
8.4s <-  DATA     /api/v1/namespaces/demo/pods  q="watch=1&resourceVersion=8199"
8.4s ->  REQUEST  /api/v1/namespaces/demo/pods  q="watch=1&resourceVersion=8207"
8.4s <-  DATA     /api/v1/namespaces/demo/pods  q="watch=1&resourceVersion=8199"
8.5s ->  CLOSE    /api/v1/namespaces/demo/pods  q="watch=1&resourceVersion=8199"
8.5s <-  STATUS   /api/v1/namespaces/demo/pods  q=""   {"state":"closed"}

Nothing after that but pings, so nothing tries to reconnect. The second REQUEST reuses the first connection instead of getting its own, the DATA after it is tagged with the old query so the frontend can't route it (createWrapperMessage uses conn.Query), and the CLOSE takes down the connection the new subscriber is on. The "closed" notice goes out with no query at all (writeStatusLocked), so the frontend never learns its watch died.

The query does reach the API server fine (L408). It's only dropped when the connection gets filed in m.connections for later lookup.

I checked this two ways: a backend test with a fake API server, and a real minikube cluster.

1. Backend test, no cluster needed

Sends two REQUESTs that differ only in query, closes the first, checks the second still gets frames. The fake cluster pushes a frame on its own rather than echoing, since that's what a real watch does. All four assertions fail:

2. Real cluster

minikube, with a namespace to watch:

apiVersion: v1
kind: Namespace
metadata:
  name: demo
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: demo
  namespace: demo
spec:
  replicas: 2
  selector:
    matchLabels: { app: demo }
  template:
    metadata:
      labels: { app: demo }
    spec:
      containers:
        - name: pause
          image: registry.k8s.io/pause:3.9

Then:

1. cd frontend && REACT_APP_ENABLE_WEBSOCKET_MULTIPLEXER=true npm start
2. open http://localhost:3000/c/minikube/pods?namespace=demo
3. kubectl delete pod <one of the two> -n demo

Before the delete, two pods running:

25 seconds after. kubectl already had the replacement pod by then, the UI is still showing the deleted one stuck at "Terminating":

Same cluster, same delete, flag off, one second later:

A kubectl scale deployment demo -n demo --replicas=3 afterwards doesn't show up either with the flag on, and lands immediately with it off.

Flag on, after the scale (still showing the original two):

Flag off, baseline and after the scale:

Same two-event test opens 19 websockets with the multiplexer off and 2 with it on, so the feature is doing what it's meant to. It just can't tell two subscriptions apart once they share a path.

Possible fix

Add query to the key and pass it at the five call sites (L428, L609, L770, L1065, L1113):

return clusterID + ":" + path + ":" + userID + ":" + query

The frontend already sends query on both REQUEST and CLOSE, so nothing changes there. It leaves a new upstream connection per event, which is wasteful but not incorrect, and is what the non-multiplexer path already does.

How this sits with #7447

#7462 takes resourceVersion out of the watch identity, so once it lands this particular reproduction stops firing. The key would still be wrong: a list and an open details drawer watch the same path with different queries, so closing the drawer should still close the list's connection. I couldn't demonstrate that one cleanly since the list freezes from the first event either way on current main, so I'm flagging it as reasoning from the code rather than something I reproduced.

Both look worth fixing and they don't overlap. #7462 stops the churn, this stops the collision.

Happy to send a PR, the test is ready to go with it. Can also attach the full run logs and raw test output.

Environment

• Installation type: dev build from main @ 03e604ae8
• Headlamp Version: frontend 0.45.0
• Kubernetes: minikube v1.39.0, k8s v1.37.0, vfkit driver on macOS arm64

Are you able to fix this issue?

Yes (I will propose a PR).