MQTT client can permanently stop reconnecting after the broker becomes unreachable

Author: hkraalCreated Sep 17, 2026Updated Sep 18, 2026

Labelsproblem

What happened?

This issue is a follow-up on #31864 which hits multiple users periodically. Lacking experience in Typescript but having the drive to get this issue solved I tasked Claude with finding a minimalistic fix. Below is Claude's rationale with typical "LLMsplaining" which I tried to keep (somewhat) to the point.

I'll open a PR shortly for a fix, this issue is intended as followup + the how and the why.

When the MQTT broker becomes unreachable, Zigbee2MQTT can reach a state where it never reconnects, not even after the broker returns. Only a restart recovers it. Zigbee2MQTT keeps running and keeps logging Not connected to MQTT server! every 10 seconds, but makes no further connection attempt.

The cause is in the mqtt (MQTT.js) client, which latches into a state where its own auto-reconnect is permanently disabled. Zigbee2MQTT does not notice: lib/mqtt.ts connects once (lib/controller.ts:169) and then relies entirely on MQTT.js auto-reconnect, and its 10s connectionTimer only logs.

What did you expect to happen?

That Zigbee2MQTT keeps retrying until the broker is reachable again.

How to reproduce it (minimal and precise)

Real setup:

Run the MQTT broker on a different host than Zigbee2MQTT.
Let Zigbee2MQTT connect.
Make the broker unreachable without a TCP reset, so packets are dropped rather than refused: iptables -I INPUT -p tcp --dport 1883 -j DROP on the broker host, or drop the tunnel. A docker stop sends a RST and does not trigger this.
Wait a few minutes, then restore reachability.
Zigbee2MQTT never reconnects; the log only repeats Not connected to MQTT server!.

Standalone reproduction, no Zigbee hardware or broker needed. Exits 1 on the bug, 0 when reconnect works:

npm i [email protected] mqtt-packet && node repro.js

javascript

// Simulates a broker host that silently disappears (dead tunnel / frozen VM): the TCP socket
// stays open but nothing is answered, and socket.destroy() does not complete immediately
// because a write is still pending.
const net = require("node:net");
const mqttPacket = require("mqtt-packet");
const {connectAsync} = require("mqtt");

const PORT = 18830;
const ts = () => new Date().toISOString().substring(11, 23);
const log = (...a) => console.log(ts(), ...a);
const sleep = (ms) => new Promise((r) => setTimeout(r, ms));

let blackhole = false;

const server = net.createServer((socket) => {
    const parser = mqttPacket.parser({protocolVersion: 4});
    socket.on("error", () => {});
    parser.on("error", () => {});
    parser.on("packet", (packet) => {
        if (blackhole) return;
        if (packet.cmd === "connect") socket.write(mqttPacket.generate({cmd: "connack", returnCode: 0, sessionPresent: false}));
        else if (packet.cmd === "subscribe")
            socket.write(mqttPacket.generate({cmd: "suback", messageId: packet.messageId, granted: packet.subscriptions.map(() => 0)}));
        else if (packet.cmd === "publish" && packet.qos === 1) socket.write(mqttPacket.generate({cmd: "puback", messageId: packet.messageId}));
        else if (packet.cmd === "pingreq") socket.write(mqttPacket.generate({cmd: "pingresp"}));
    });
    socket.on("data", (d) => parser.parse(d));
});

(async () => {
    await new Promise((r) => server.listen(PORT, "127.0.0.1", r));

    // keepalive shortened from zigbee2mqtt's default 60 to speed the test up
    const client = await connectAsync(`mqtt://127.0.0.1:${PORT}`, {keepalive: 5});
    let attempts = 0;
    client.on("error", (e) => log("MQTT error:", e.message));
    client.on("reconnect", () => log(`>> reconnect attempt #${++attempts}  connected=${client.connected}`));

    // zigbee2mqtt's connection check (lib/mqtt.ts), logs only
    setInterval(() => {
        if (!client.connected) {
            log(
                `Not connected to MQTT server!  connected=${client.connected} reconnecting=${client.reconnecting} ` +
                    `disconnecting=${client.disconnecting} disconnected=${client.disconnected} ` +
                    `reconnectTimer=${client.reconnectTimer ? "armed" : "null"} attempts=${attempts}`,
            );
        }
    }, 5000);

    await sleep(1000);

    // A pending write on a black-holed socket makes destroy() complete late. Simulated here so the
    // test is deterministic; on a real dead tunnel the kernel does this for you.
    const sock = client.stream;
    const origDestroy = sock.destroy.bind(sock);
    sock.destroy = (...args) => {
        setTimeout(() => origDestroy(...args), 1500);
        return sock;
    };

    log("### broker host disappears (no answers, socket stays open)");
    blackhole = true;

    await sleep(40000);
    log("### broker host is BACK");
    blackhole = false;

    await sleep(40000);
    log(`### RESULT: connected=${client.connected} attempts=${attempts} -- expected: reconnected`);
    process.exit(client.connected ? 0 : 1);
})();

Reproduced on [email protected] (shipped in the 2.14.1 image) and [email protected] (what ^5.15.2 resolves to today), on Node 22.

Zigbee2MQTT version

2.14.1

Adapter firmware version

Not relevant — this is in the MQTT client, independent of the Zigbee adapter.

Adapter

Not relevant — this is in the MQTT client, independent of the Zigbee adapter.

Setup

Docker container running on a raspberry Pi, the MQTT broker is hosted externally and reachable via a WireGuard tunnel.

Device `database.db` entry

Not applicable.

Debug log

Output of the standalone reproduction above, on [email protected] / Node 22, repeated identical lines elided:

mqtt 5.16.0 / node v22.23.2
17:25:11.129 ### broker host disappears (no answers, socket stays open)
17:25:17.644 MQTT error: Keepalive timeout
17:25:18.651 >> reconnect attempt #1  connected=true
17:25:20.140 Not connected to MQTT server!  connected=false reconnecting=true disconnecting=true disconnected=true reconnectTimer=null attempts=1
17:25:25.149 Not connected to MQTT server!  connected=false reconnecting=true disconnecting=true disconnected=true reconnectTimer=null attempts=1
17:25:49.158 MQTT error: connack timeout
17:25:51.144 ### broker host is BACK
17:25:55.195 Not connected to MQTT server!  connected=false reconnecting=true disconnecting=true disconnected=true reconnectTimer=null attempts=1
...
17:26:30.260 Not connected to MQTT server!  connected=false reconnecting=true disconnecting=true disconnected=true reconnectTimer=null attempts=1
17:26:31.151 ### RESULT: connected=false attempts=1 -- expected: reconnected

Two lines matter:

17:25:18.651 — connected is still true, so _reconnect() takes the end() branch instead of reconnecting
from 17:25:20 — disconnecting=true and reconnectTimer=null, and they stay that way. No further attempt is made, including after the broker returns at 17:25:51.

Notes

The latch belongs upstream in MQTT.js. Candidates there:

_cleanUp(forced) should set this.connected = false synchronously instead of waiting for the stream's close; that alone closes the race
_reconnect()'s this.end(() => this.connect()) branch leaves disconnecting latched whenever reconnecting is true; connect()'s reset condition does not cover that case
connackTimer is a single shared field across overlapping streams, so a late close from an orphaned socket can clear the live stream's only timeout

[email protected] addresses none of these, and lib/mqtt.ts is unchanged on dev with the dependency still at ^5.15.2, so a Zigbee2MQTT-side guard may be worthwhile regardless. A minimal one, tested against the compiled dist/mqtt.js in the 2.14.1 image:

typescript

// in the existing 10s connectionTimer in lib/mqtt.ts, inside `if (!this.isConnected())`
if (this.client.disconnecting) {
    logger.warning("Forcing reconnect to MQTT server");

    this.client.disconnecting = false;
    this.client.reconnect();
}

disconnecting is the latch itself, so no extra state or counter is needed, and the branch is only reached in a state where Zigbee2MQTT is already permanently disconnected.

Clearing the flag first is required. In the latched state disconnecting === true while disconnected is still undefined, because end() never completed — it is parked on this.once('outgoingEmpty', …) waiting for a QoS 1 ack that can no longer arrive (Zigbee2MQTT's retained bridge/state republish sits in outgoing). reconnect() then hits