Node stuck in graceful shutdown keeps the leader lock, blocking failover of leader-only work

Problem

A node that hangs during graceful shutdown keeps the leader lock forever. No other node can take over, so leader-only work has no way to fail over for as long as the process stays alive.

Observed case

Two node 7.2.0-beta.2 test cluster. The leader received SIGTERM and was still shutting down 50 minutes later. The heartbeat stays fresh because NodePingThread.stopOnGracefulShutdown() returns false by design, so the node keeps advertising itself as leader while it is dying.

What made the hang unbounded: the search cluster was in flood stage with indices set to read_only_allow_delete. Writes were rejected, so the output buffer backed up, the process buffer filled, the input buffer filled, and the inputs could not finish stopping. The shutdown never got past inputSetupService.awaitTerminated().

Proposed fix

Release the leader lock at the serverStatus.shutdown() transition, before the REST API and the inputs are stopped. At that point the node has already set lb_status to DEAD and has committed to dying. Everything after is best effort draining.

The handover path already exists and does not need to be built:

1. Releasing the lock posts LeaderChangedEvent.
2. PeriodicalsService.leaderChanged() sees isLeader() == false and calls stopPeriodicals(runningLeaderNodePeriodicals) on the dying node.
3. The next node to acquire the lock starts them.

Risk analysis

Risk of leaving it: medium to high. Risk of fixing it: low.

It is rare, but it happens during outages and rolling restarts rather than at random, so it shows up when the cluster is already in trouble. Once it happens nothing recovers on its own and someone has to kill the process by hand.

The fix reuses a handover that already happens today, so it is a small change with one hazard that is easy to catch in review.

Companion change

Put a deadline on the three unbounded awaitTerminated() calls in GracefulShutdown. A node that has told the load balancer it is dead should have a hard limit on how long it is allowed to take to actually die. This bounds every consequence of a stuck shutdown rather than just the leadership one, and the two changes are independent.

How to reproduce

1. Start a two node cluster.
2. Fill the search cluster disk past the flood stage watermark so indices go read_only_allow_delete.
3. Send enough traffic to fill the input buffer on the leader.
4. Send SIGTERM to the leader.
5. The leader hangs in shutdown, keeps is_leader: true, and the other node never takes over.