MigrationController marks Allocated GameServers Unhealthy on transient Node address mismatch

What happened:

On production AKS clusters running Agones 1.57.0, multiple Allocated GameServers hosting active game sessions were simultaneously marked Unhealthy by the migration-controller and their Pods were terminated.

For directly correlated cases, the Kubernetes Warning event:

• reportingController: migration-controller
• reason: Unhealthy
• message: Node migration occurred

was emitted 0–2 seconds before the affected Pod received SIGTERM.

The affected nodes remained Ready and schedulable. The Kubernetes scheduler continued binding new GameServer Pods to the same nodes immediately after the incident.

In a representative incident:

• the GameServer address at allocation time matched the Node public IP;
• the same public IP was still present before and after the termination;
• no persistent IP change was observed;
• no NodeNotReady event occurred;
• no nearby WireServer or IMDS error was observed.

This suggests that Node.status.addresses may have been temporarily incomplete, or observed as incomplete through the informer cache. A single address mismatch was then interpreted as a real Node migration.

What you expected to happen:

A transient address-only mismatch should not immediately mark an Allocated GameServer as Unhealthy, especially when:

• pod.spec.nodeName still matches gameserver.status.nodeName;
• the Node UID has not changed;
• the Node remains Ready and schedulable.

The controller should revalidate the mismatch after a grace period or require multiple consecutive mismatches before terminating an Allocated GameServer.

Actual Node replacement or migration should still be detected and handled.

How to reproduce it (as minimally and precisely as possible):

This can be reproduced at the controller/unit-test level:

1. Create a Ready Node with an address.
2. Create a GameServer Pod assigned to that Node.
3. Set the GameServer state to Allocated, with status.nodeName and status.address matching the Node.
4. Temporarily remove the GameServer address from Node.status.addresses, while keeping the same Node name, UID and Ready condition.
5. Trigger a Pod update or informer resync.
6. Observe that the migration-controller treats it as a migration and changes the GameServer state to Unhealthy.
7. Restore the Node address.

The restoration is too late because the GameServerSet has already begun deleting the GameServer.

An actual Pod API update may not be required because the shared informer has a 30-second resync period and the migration-controller handles synthetic Pod Update events.

Anything else we need to know?:

This appears to be a remaining gap after:

• https://github.com/agones-dev/agones/issues/3115
• https://github.com/agones-dev/agones/pull/3116

PR #3116 made the controller tolerate any matching address on the Node. However, if the previously assigned address is temporarily absent from the entire address list, the current logic still immediately marks an Allocated GameServer as Unhealthy.

Relevant code paths:

• isMigratingGameServerPod() checks whether any Node address matches the GameServer address.
• syncGameServer() immediately sets Ready/Reserved/Allocated GameServers to Unhealthy on a mismatch.
• There is no debounce, grace period, consecutive-failure check, Node Ready check, or special protection for Allocated GameServers.

Possible mitigation options:

• requeue and revalidate address-only mismatches after a configurable grace period;
• require two or more consecutive mismatches;
• use Node UID changes as a stronger migration signal;
• avoid marking Allocated GameServers Unhealthy for a single address-only mismatch while the same Node remains Ready;
• expose a metric/event containing the GameServer address and observed Node addresses.

Sanitized event sequence:

T0: migration-controller Warning event, reason=Unhealthy,
    message="Node migration occurred"
T0 to T0+2s: affected Pod receives SIGTERM
Shortly afterward: scheduler binds replacement Pods to the same Node

Environment:

• Agones version: 1.57.0
• Kubernetes version: 1.33.6
• Cloud provider or hardware configuration: Azure Kubernetes Service
• Install method: Helm
• Troubleshooting guide log(s): Kubernetes migration-controller events and application SIGTERM logs
• Others: The same immediate-Unhealthy behavior is still present in the current main branch