[Bug]: Matchmaker, failed health check and hidden rooms

Context

On our game, we have a long history of players noticing their room is sometimes no longer visible, and the known workaround is to leave and rejoin the room to make it visible again for other players. Another workaround is to tweak the room options, which trigger setMetadata and re-add the room in matchmaker listing

@brobinett worked on this topic and made some interesting observations: https://github.com/keldaanCommunity/pokemonAutoChess/pull/4126

The issue: A room and its listing have separate lifetimes. When a process misses a single IPC health-check ping, @colyseus/core's MatchMaker.healthCheckProcessId calls removeRoomsByProcessId and RedisDriver.cleanup HDELs the roomcaches entry of every room on that process. The rooms keep running; only their listings are gone, and nothing ever puts them back.

We noticed these hidden lobbies often happened "on the hour": 12:00, 13:00... the hypothesis is that some cron jobs are running at that moment and cause a latency spike for a few seconds. If the health check happens at that precise moment and fails, the room is gone.

Bug description

The current workaround @brobinett found is a check at regular interval to manually re-add the room to the listing. But i don't think it is something that should be done on our side. My suggestion would be that in case of a failed health check, Colyseus matchmaker performs at least two other retries after a few seconds before declaring the room dead and unlisting it.

Reproduction

@brobinett mentionned they manage to reproduce it on two processes with Redis by stalling one of these processes.

Steps to reproduce

I don't know the details on how he manage to force the stalling of one process. Maybe with an infinite loop ? I suppose it is easy to simulate a failing health-check though.

Environment & Versions

Colyseus 0.17.10 Node 24 we run the game on 8 threads, 4 machines