[Core] After GCS failover, raylet grants lease requests queued by the previous GCS; worker stays ray::IDLE and leaks the node's GPU

What happened + What you expected to happen

Summary: after a GCS failover, a worker raylet still holds lease requests that the old GCS queued on it. When the node's resource later frees up, the raylet grants that stale lease to a fresh worker process. The reply goes to a GCS that no longer exists, so nobody ever runs anything on that worker. It sits as ray::IDLE forever, holding the resource (in our case the node's only GPU). The new GCS cannot schedule anything needing that resource on the node, and nothing ever cleans it up.

How we hit it (Ray 2.57.0, KubeRay RayService with GCS FT, Ray Serve on 1-GPU nodes):

1. The Serve autoscaler asked for a second replica while the node's only GPU was busy. GCS queued the lease on that raylet (node_manager.cc: The lease with ID ... cannot be scheduled right now).
2. The head pod died. A new head came up about a minute later and the raylets reconnected. The raylet kept the queued lease.
3. About a minute after that, Serve stopped the replica that held the GPU (KillLocalActor).
4. 7 ms later the raylet granted the stale lease: worker_pool.cc:531: Started worker process with pid 919. That worker never received a task and stayed ray::IDLE. debug_state.txt lists the lease under Resource usage with {GPU: 1, CPU: 2}.
5. From then on the new GCS logs Failed to lease worker from node ... as the resources are not enough for that node, ray status shows the GPU in use, nvidia-smi shows 0 MiB, and the autoscaler adds new GPU nodes instead. The new GCS later destroyed the actor the lease was for, but since it never issued that lease, nothing was cancelled on the raylet.

kill -9 on the idle worker pid frees the GPU within a second, which confirms the allocation is tied only to the orphaned lease.

Expected: on GCS failover the raylet should drop (or the new GCS should reconcile) lease requests issued by the previous GCS incarnation, so a stale lease can never be granted to a client that no longer exists.

Related: #61072 (GCS-side leasing state cleanup) and #63763 (zombie actors after GCS restart). Neither covers the raylet keeping queued lease requests across GCS failover.

Versions / Dependencies

Observed on Ray 2.57.0. Reproduced below on rayproject/ray:2.58.0-py313 (also on 2.57.0). Python 3.13, Docker on macOS (arm64). No GPU needed.

Reproduction script

Head + one worker + Redis in Docker. A custom resource GPUx: 1 stands in for the GPU. Save as repro.sh and run it; the whole thing takes about two minutes.

#!/bin/bash
# Reproduces: raylet grants a lease that the previous GCS queued, leaking the resource.
set -e
IMG=rayproject/ray:2.58.0-py313
docker rm -f rr-redis rr-head rr-worker 2>/dev/null; docker network create rr 2>/dev/null || true
COMMON="--network rr -v $PWD:/app -e RAY_REDIS_ADDRESS=rr-redis:6379 -e RAY_external_storage_namespace=rr1 -e RAY_gcs_rpc_server_reconnect_timeout_s=600"
docker run -d --name rr-redis --network rr redis:7-alpine
docker run -d --name rr-head $COMMON $IMG ray start --head --block --num-cpus=1 --port=6379 --disable-usage-stats
docker run -d --name rr-worker $COMMON $IMG ray start --block --address=rr-head:6379 --num-cpus=2 --resources='{"GPUx": 1}' --disable-usage-stats
sleep 20

# Step 1: actor A takes the worker's only GPUx. Actor B asks for another one and stays pending,
# so the GCS queues B's lease request on the worker raylet.
docker exec -i rr-head python - <<'PY'
import ray, time
ray.init(address="auto", namespace="repro", log_to_driver=False)

@ray.remote(num_cpus=1, resources={"GPUx": 1})
class Holder:
    def ping(self): return "ok"

a = Holder.options(name="A", lifetime="detached").remote()
print("A ready:", ray.get(a.ping.remote()))
b = Holder.options(name="B", lifetime="detached").remote()   # pending forever: no GPUx left
time.sleep(15)   # give the raylet time to log "The lease with ID ... cannot be scheduled right now"
PY
docker exec rr-worker grep -c "cannot be scheduled" /tmp/ray/session_latest/logs/raylet.out || echo "lease not queued yet?"

# Step 2: GCS failover. The new GCS marks B dead (its owner died with the old head),
# but the worker raylet still has B's lease request queued.
docker kill rr-head; sleep 5; docker start rr-head; sleep 25

# Step 3: free the GPUx by killing A. The raylet immediately hands the stale lease to a new
# worker, which stays ray::IDLE and keeps GPUx. A new actor C that needs GPUx never starts.
docker exec -i rr-head python - <<'PY'
import ray, time
ray.init(address="auto", namespace="repro", log_to_driver=False)
ray.kill(ray.get_actor("A"))
time.sleep(10)
print("available after killing A:", ray.available_resources())   # no GPUx

@ray.remote(num_cpus=1, resources={"GPUx": 1})
class Holder:
    def ping(self): return "ok"

c = Holder.options(name="C").remote()
try:
    print("C ready:", ray.get(c.ping.remote(), timeout=20))
except ray.exceptions.GetTimeoutError:
    print("C is still pending after 20s")
PY

echo "--- idle worker holding the lease:"
docker exec rr-worker ps -eo pid,etime,args | grep "ray::IDLE"
docker exec rr-worker grep -A2 "Resource usage" /tmp/ray/session_latest/logs/debug_state.txt
docker exec rr-worker grep -h "Received KillLocalActor\|Started worker process" /tmp/ray/session_latest/logs/raylet.out | tail -2

Output on 2.58.0:

A ready: ok
1
available after killing A: {'CPU': 2.0, 'memory': ..., 'node:172.20.0.3': 1.0, 'node:172.20.0.4': 1.0, ...}   <- no GPUx
C is still pending after 20s
--- idle worker holding the lease:
  259       00:31 ray::IDLE
Resource usage {
    - (language=PYTHON actor_or_taskHolder.__init__ pid=259 worker_id=73dcc28b...): {CPU: 1, GPUx: 1}
}
[2026-09-15 23:18:40 I 43 43] (raylet) node_manager.cc:3626: Received KillLocalActor RPC worker_id=cd5c0978...
[2026-09-15 23:18:40 I 43 43] (raylet) worker_pool.cc:531: Started worker process with pid 259 worker_id=73dcc28b...

ray list actors at this point shows A DEAD, B DEAD, C PENDING_CREATION.

Not GPU specific. The same script with no custom resource at all (head --num-cpus=0, worker --num-cpus=1, actors @ray.remote(num_cpus=1)) leaks the worker's only CPU the same way: the idle worker holds {CPU: 1} and ray status shows 1.0/1.0 CPU used with nothing running. The raylet is keeping a queued lease request, and a lease request holds whatever resources it asked for. We noticed it with GPUs because one leaked lease takes a whole GPU node out of service, while a leaked CPU on a large node goes unnoticed. docker exec rr-worker kill -9 259 brings GPUx: 1.0 back in ray.available_resources() within a few seconds.

Issue Severity

High. Severity seems high because we have noticed a lot of essentially zombied GPUs in our clusters that need to be manually killed. Every head restart in a GCS FT cluster can permanently leak a GPU per node until someone kills the idle worker by hand or deletes the node.