[Bug]: Agent freezes permanently in the disconnected state after a late i/o timeout, with no retry and no sensor timeout involved

Welcome!

• I have read the Documentation
• I have checked the Common Issues Guide and my problem was not mentioned there.
• I have searched open and closed issues and my problem was not mentioned before.
• I have verified I am using the latest version available. You can check the latest release here.

Component

Agent

Problem Description

Two macOS agents on 0.19.0, on separate machines and on different days, permanently stopped communicating with the hub. In both cases the process stayed alive with the same PID, launchctl list reported the job healthy, and the agent never wrote another log line. launchctl kickstart -k recovered both instantly, and both reconnected within seconds.

The freeze signature is identical on both hosts, and is three lines long:

WARN Connection closed err="connection closed, code=1000, reason="
WARN Disconnected from hub
WARN Connection closed err="read tcp <agent>:<port>-><hub>:8090: i/o timeout"
[silence forever]

Two things stand out:

1. The freeze happens while the agent is already disconnected, not after a reconnect. After Disconnected from hub there is no WebSocket connection failed retry line and no Starting SSH server fallback — the agent simply never tries again. It is neither connected to the hub nor listening on 45876; I confirmed with lsof that the frozen process held no listening socket and no established connection to the hub.

2. The final Connection closed arrives ~70s after the disconnect, on a socket that was already reported closed:

• Host A: 16:32:58 disconnect → 16:34:12 i/o timeout (74s)
• Host B: 09:08:19 disconnect → 09:09:30 i/o timeout (71s)

That interval lines up with the 70s WebSocket deadline discussed in #2294. It looks like the read deadline on the old connection expires after the disconnect has already been handled, and that late close is what wedges the reconnect loop — but that is my inference from the logs, not something I can prove from outside.

This is not the temperature-sensor path. Host A has exactly one temperature collection timed out line in its entire log history, from 2026-06-21, roughly three months before the freeze. Host B has zero, ever. Neither has SENSORS or SENSORS_TIMEOUT set. So whatever wedges the agent here does not require a sensor timeout to trigger it.

Related issues, and how this differs:

• #2280 — macOS agent hangs permanently, no crash, no reconnect. Closest match, but there every freeze is preceded by a temperature-sensor timeout, and the last line is Stopping SSH server immediately after a successful WebSocket connected. Here there is no sensor timeout, no successful reconnect, and no SSH fallback — the agent freezes in the disconnected state instead of the reconnected one. Possibly the same underlying wedge reached by a different path.
• #2294 (closed) — 70s WebSocket deadline causing reconnect loops. Same 70s number, opposite outcome: that one loops, this one stops.
• #2058 (closed) — websocket does not retry after failure, on 0.18.7. Same end state, but that log shows repeated retry attempts before giving up; mine shows none at all.

The practical consequence is that this fails silently in both directions. Host A was dark for 2 days (2026-09-09 16:34 → 2026-09-11 09:15) before a human noticed the dashboard. KeepAlive=true in the Homebrew plist cannot help, because the process never exits — it hangs. And the agent's own log is not a usable liveness signal either: a healthy agent on this fleet went two full days without writing a line, so "no recent log output" does not distinguish healthy from frozen.

The hub was healthy throughout: monitor container up 37h, serving HTTP 200, other agents reporting normally. The hub restarted at 2026-09-10 03:07 and 03:08, in the middle of Host A's outage, and Host A did not reconnect even then.

Expected Behavior

After Disconnected from hub, the agent should keep attempting to reconnect indefinitely, or exit so the service manager restarts it. A late timeout on an already-closed connection should not be able to permanently stop the reconnect loop.

Failing that, it would help a great deal if a wedged agent were detectable from outside — either by exiting, or by continuing to log its retry attempts, so that a watchdog has something to key on.

Steps to Reproduce

I cannot reproduce this on demand. It occurred twice in three days across two hosts. Both times the trigger appears to be a clean hub-side close (code=1000) followed by the old connection's read deadline expiring ~70s later.

1. Run the agent on macOS via Homebrew in WebSocket-push mode (HUB_URL + TOKEN).
2. Leave it running for days.
3. Watch for a code=1000 close followed ~70s later by a second Connection closed ... i/o timeout.
4. Check whether any further log line is ever written.

OS / Architecture

darwin/arm64, macOS 26.6.2, on both hosts (Apple M1 Ultra and Apple M4 Max)

Beszel version

Agent 0.19.0 (Homebrew), Hub 0.19.0 (henrygd/beszel:latest, Docker)

Installation method

Homebrew (agent), Docker (hub)

Configuration

# agent, both hosts — no SENSORS or SENSORS_TIMEOUT override
LISTEN=45876
HUB_URL=http://192.168.4.135:8090   # plain HTTP over LAN, no proxy, no tunnel
TOKEN=<redacted>
KEY=<redacted>
GPU_COLLECTOR=macmon
PATH=/opt/homebrew/bin:/usr/local/bin:/usr/bin:/usr/sbin:/bin

Hub Logs

2026/09/10 03:07:40 Server started at http://0.0.0.0:8090
2026/09/10 03:08:30 Server started at http://0.0.0.0:8090

Nothing else logged during either outage.

Agent Logs

--- Host A (Apple M1 Ultra) — frozen 2026-09-09 16:34 until manual restart 2026-09-11 09:15 ---
2026/09/07 10:09:46 INFO Detected network interface name=en1 sent=1925541700 recv=1109725311
2026/09/09 16:32:58 WARN Connection closed err="connection closed, code=1000, reason="
2026/09/09 16:32:58 WARN Disconnected from hub
2026/09/09 16:34:12 WARN Connection closed err="read tcp 192.168.4.37:63248->192.168.4.135:8090: i/o timeout"
[no further log lines for 1d 16h, until manual launchctl kickstart]
2026/09/11 09:15:16 INFO WebSocket connected host=192.168.4.135:8090

--- Host B (Apple M4 Max) — frozen 2026-09-11 09:09 until manual restart 2026-09-11 09:15 ---
2026/09/09 20:08:38 INFO WebSocket connected host=192.168.4.135:8090
2026/09/09 20:08:38 INFO Stopping SSH server
2026/09/11 09:08:19 WARN Connection closed err="connection closed, code=1000, reason="
2026/09/11 09:08:19 WARN Disconnected from hub
2026/09/11 09:09:30 WARN Connection closed err="read tcp 192.168.4.24:49865->192.168.4.135:8090: i/o timeout"
[no further log lines until manual launchctl kickstart]
2026/09/11 09:15:20 INFO Shutting down cause="terminated signal received"
2026/09/11 09:15:20 INFO WebSocket connected host=192.168.4.135:8090