ThreadJobExecutor.aclose() hangs when the runner exits before its ack is read

Author: pabloFuenteCreated Sep 11, 2026Updated Sep 11, 2026

Labelsbug

Bug Description

With JobExecutorType.THREAD, a graceful shutdown of a worker that has idle prewarmed runners logs job did not ack shutdown in time for most of them and then never exits. drain() has already returned by then, so drain_timeout doesn't help. Only a second signal ends the process. On a 16-core host (so 16 idle runners) I've been consistently reproducing this on every shutdown.

Expected Behavior

A worker running with JobExecutorType.THREAD with idle prewarmed runners should allow graceful shutdowns, just like JobExecutorType.PROCESS does.

Reproduction Steps

The minimal worker below is a proof of the bug: THREAD executor, 32 idle runners, no LiveKit server needed, SIGTERM once the runners are up. On main 27 of the 32 runners log did not ack and the process never exits. With the proposed solution it exits in about a second with no errors.

python

import logging
import sys
from livekit.agents import AgentServer, JobContext, JobExecutorType, cli
from livekit.agents.worker import ServerType

logging.basicConfig(level=logging.INFO)

async def entrypoint(ctx: JobContext) -> None:
    pass

if __name__ == "__main__":
    server = AgentServer(
        job_executor_type=JobExecutorType.THREAD,
        num_idle_processes=32,  # the more idle runners, the more likely one loses the race
        ws_url="ws://127.0.0.1:7880",  # nothing listens here, on purpose
        api_key="devkey",
        api_secret="secret",
        max_retry=sys.maxsize,  # keep retrying the connection instead of giving up
    )
    server.rtc_session(type=ServerType.ROOM, agent_name="repro")(entrypoint)
    cli.run_app(server)

bash

uv run python repro.py start  # wait until "job runner initialized" was logged 32 times
kill -TERM <pid>              # or Ctrl+C

You will get an output log like:

log

{"message": "draining worker", "level": "INFO", "name": "livekit.agents", "id": "unregistered", "timeout": 3600, "timestamp": "2026-09-11T11:48:27.159891+00:00"}
INFO:livekit.agents:shutting down worker
{"message": "shutting down worker", "level": "INFO", "name": "livekit.agents", "id": "unregistered", "timestamp": "2026-09-11T11:48:27.160304+00:00"}
ERROR:livekit.agents:job did not ack shutdown in time
{"message": "job did not ack shutdown in time", "level": "ERROR", "name": "livekit.agents", "tid": 164226, "timestamp": "2026-09-11T11:48:37.162348+00:00"}
ERROR:livekit.agents:job did not ack shutdown in time
{"message": "job did not ack shutdown in time", "level": "ERROR", "name": "livekit.agents", "tid": 164227, "timestamp": "2026-09-11T11:48:37.163767+00:00"}
ERROR:livekit.agents:job did not ack shutdown in time
{"message": "job did not ack shutdown in time", "level": "ERROR", "name": "livekit.agents", "tid": 164228, "timestamp": "2026-09-11T11:48:37.163890+00:00"}
ERROR:livekit.agents:job did not ack shutdown in time
{"message": "job did not ack shutdown in time", "level": "ERROR", "name": "livekit.agents", "tid": 164229, "timestamp": "2026-09-11T11:48:37.163980+00:00"}
...

Operating System

Ubuntu 26.04

Package Versions

bash

livekit-agents==1.8.1

Proposed Solution

See https://github.com/livekit/agents/pull/7225

Source: livekit/agents

View original on GitHub View discussion on GitHub