[bug]: [wtclient] catch-up after a prolonged tower outage starves a running node — forwarding stops, gRPC hangs, one Postgres txn left open

Pre-Submission Checklist

• I have searched the existing issues and believe this is a new bug.
• I am not asking a question about how to use lnd, but reporting a bug (otherwise open a discussion).

LND Version

v0.21.1-beta (both the client node and the watchtower server node)

LND Configuration

Client:

wtclient.active=true
wtclient.sweep-fee-rate=100
db.backend=postgres
db.postgres.maxconnections=25
db.postgres.timeout=0
db.postgres.channeldb-with-global-lock=false

wtclient.db was migrated into Postgres in Dec 2025 (use-native-sql=true), so the "delete wtclient.db and restart" workaround from older threads is not available.

Tower (separate host):

watchtower.active=true
watchtower.listen=0.0.0.0:9911

Backend Version

bitcoind (client node is fully synced throughout; the backend is not implicated)

OS/Distribution

Ubuntu, x86_64, 8 vCPU / 16 GB, lnd under systemd

Bug Details & Steps to Reproduce

Two phases. Phase 1 is the known failure; phase 2 is what I think is new.

Phase 1 — tower rejects everything for a long period. The tower rejected every state update from this client with CodePermanentFailure and StateUpdateCodeSeqNumOutOfOrder. Client-side seqnums were frozen; WTCL=debug showed Unable to get new tower candidate ... reason: exhausted all tower candidates on every client type. Duration is unknown — it predates log retention on both hosts. lncli wtclient towers, stats and remove all hung during this period. This part matches #6397 / #4420 / #6785.

Restarting lnd on the tower (not the client) cleared it. Within ~20 minutes each client type negotiated a fresh session and uploads resumed with code=0 acks and in-order seqnums.

Phase 2 — the catch-up then took the node down. The client began a sustained catch-up and held it for ~3 hours:

• ~100 state updates/min accepted, flat, no taper
• one new session acquired per ~10 min (consistent with the 1024-updates-per-session cap in #7045)
• sessions(acquired=…) went 202 → 219, exhausted 205 → 222 over that window
• meanwhile the node's own live backup demand was ~7 tasks/hour (tasks(received=255) total across 36 h of uptime, against accepted=226526)

After ~3 hours the node had effectively stopped working as a routing node, while systemctl still reported lnd active:

The last three rows are the interesting part: lnd was not blocked waiting on the database. Nothing was executing and no backend was waiting on a lock — lnd had stopped issuing work, while leaving one transaction open. With db.postgres.timeout=0 nothing ever times out, so the condition does not self-clear.

Resolution: set wtclient.active=false, restart lnd. Immediately after:

Expected Behavior

A watchtower backlog is not urgent work — the node is online and self-watching. It should be paced or backgrounded such that it cannot starve the htlcswitch, the peer subsystem and the RPC server. At minimum, an unbounded catch-up should not be able to leave a Postgres transaction open indefinitely while the client makes no progress elsewhere.

Debug Information

Sanitised log shapes (client):

[INF] WTCL: (anchor) Client stats: tasks(received=255 accepted=226526 ineligible=0) sessions(acquired=219 exhausted=222)

Tower side, with WTWR=debug (120 s sample, IDs redacted):

[DBG] WTWR: Received MsgStateUpdate(seqnum=177 last_applied=176 is_complete=0 hint=<redacted>) from <session-key>
[DBG] WTWR: State update 177 accepted for <session-key>
[DBG] WTWR: Sending MsgStateUpdateReply(code=0 last_applied=177) to <session-key>

112 unique hints in that window, and a second window ~13 min later shared zero hints with the first — so the client was sending genuinely new states, not replaying.

Also, and this may be the more actionable half: lncli wtclient towers --include_sessions never returned — killed after 18m16s having produced no output. An earlier call was killed at 25 s. getinfo was measured at 89.7 s while that call was outstanding, and killing the client process did not restore responsiveness. This looks like #6660 (that reporter saw 10.9 GB RES and a 6m57s runtime, with the same trigger: data queued up while the tower was down). It makes the one RPC you need for diagnosis unusable exactly when you need it.

Related

• #7045 — documents the 1024-updates-per-session design and the candidate loop. Note its fix retires a session only once all its updates are acked; with a tower rejecting everything, no session can ever be retired, so the fix cannot engage.
• #7121 — closest prior match (Postgres, exhausted: 2116 vs acquired: 3, lnd+postgres spinning), also resolved by wtclient.active=false.
• #10646 — open, v0.20.1; dead tower, wtclient towers taking minutes, tower not removable.
• #6660 — lncli wtclient tower memory/latency blowup.
• #7949 — the phase-1 error; resolved upstream by wtclient session terminate (#8239).

Caveat on my own data: my exhausted-session count (222) is an order of magnitude below #7121's 2,116, so I am not certain the candidate loop alone explains the stall — the hung RPC may be carrying more of it. Happy to gather more if there is something specific worth capturing before I re-enable the client.