Telegram inbound can die silently for days: pollingLoop retries forever with no give-up, and logs nothing on success

Affected NanoClaw version or commit

v2.3.0 (observed on v2.1.54, re-verified against the adapter shipped with v2.3.0)

Host platform

Linux

What happened?

Inbound Telegram died silently for ~4 days. The host stayed active, outbound delivery kept working, and the agent's scheduled tasks still ran and delivered — so every surface an operator would check looked healthy. Only inbound was gone.

The host log showed 11,178 consecutive getUpdates failures, with the adapter's consecutiveFailures counter never resetting.

Root cause is in @chat-adapter/telegram (4.29.0), dist/index.js → pollingLoop:

async pollingLoop(config) {
  let consecutiveFailures = 0;
  const MAX_BACKOFF_MS = 3e4;
  while (this.pollingActive) {
    try {
      const updates = await this.telegramFetch("getUpdates", {...});
      consecutiveFailures = 0;
      ...
    } catch (error) {
      consecutiveFailures++;
      const backoffMs = Math.min(config.retryDelayMs * 2 ** (consecutiveFailures - 1), MAX_BACKOFF_MS);
      this.logger.warn("Telegram polling request failed", { error, retryDelayMs: backoffMs, consecutiveFailures });
      await this.sleep(backoffMs);
    }
  }
}

It retries forever with a 30s backoff ceiling: no give-up, no self-heal, and no escalation at any failure count. Nothing above it notices.

The detail that makes this hard to detect: the loop logs nothing on a successful poll — consecutiveFailures = 0 is silent. So "healthy and quiet" and "loop is dead" are byte-identical in the logs. Any health check built only on the failure stream is blind exactly when the loop stops emitting.

There is also no seam to hang a health check on. ChannelAdapter declares isConnected(), but nothing on the host ever calls it, and the Chat SDK bridge implements it as return true.

What did you expect?

Either the adapter escalates a polling loop that has been failing for minutes (so systemd's Restart=always can recover it), or NanoClaw notices that a channel's inbound path is dead and surfaces it. A 4-day silent inbound outage on a host reporting active should not be reachable.

How can we reproduce it?

1. Wire a Telegram channel and let it run normally.
2. Break outbound network reachability to api.telegram.org from the host (firewall rule, or DNS blackhole) so getUpdates fails while the process keeps running.
3. Observe: getUpdates failures accumulate in the log with a 30s backoff, forever. The host stays active, outbound delivery still works, scheduled tasks still fire. consecutiveFailures never resets and nothing ever gives up.
4. Restore reachability after the counter is high. The loop recovers, but nothing ever reported the outage — and had the loop instead stopped, step 3 would have produced no log lines at all.

OS version and CPU architecture

Ubuntu 24.04.4 LTS, x86_64 (WSL2, kernel 6.18.33.1-microsoft-standard-WSL2)

Docker version

Docker 29.5.3

Channel or interface

Telegram

Relevant redacted logs

Telegram polling request failed  {"error":"...","retryDelayMs":30000,"consecutiveFailures":11178}

(One line, repeated with a monotonically increasing counter, for ~4 days. No other channel or host error accompanied it.)

Additional context — a watchdog attempt, and why the naive version made it worse

I ran a local watchdog for this and have since removed it, but the failure mode it uncovered is worth recording, because it is a trap anyone fixing this will hit.

v1 (counter only). Bounce the poller after N consecutive failures: stopPolling() then startPolling(). This caused a second, worse outage lasting three days. A transient network blip pushed the counter to the soft threshold; stopPolling() succeeded; startPolling() then threw inside resetWebhook/deleteWebhook — still the same blip. Polling was now stopped, not failing. No further failures were ever logged, so the counter never advanced to the hard threshold and the process was never restarted. Strictly worse than the bug it was fixing, which at least kept retrying.

The lesson: a failure-counter watchdog is blind to the state it can itself create. Any bounce must re-verify that polling is actually running again rather than trusting a clean return from startPolling.

v2 (two independent signals). What actually held:

• a failure counter fed from the adapter's logger — catches a loop that is alive but not getting through (the original outage);
• a liveness probe (60s timer) reading the adapter-internal pollingActive flag — catches a loop that is gone (the self-inflicted one). The flag is true while the loop runs and is cleared both on a clean stop and by startPolling's own catch. Read structurally, so an upstream rename yields undefined = "cannot tell" = treated healthy, degrading to counter-only rather than restart-looping the process.

Plus one invariant: never leave the poller stopped. A bounce retries startPolling with backoff and re-reads pollingActive; if it still cannot get polling running it escalates to process.exit(1) and lets the service manager restart — the path that provably fixes it. One bounce per episode, and the "already bounced" flag clears only when the adapter reports a reset counter (a poll that actually succeeded), never on a "successful" bounce, or it thrashes.

Thresholds mattered: 1-2 isolated getUpdates failures are normal and must not trigger a bounce. Soft at 10 (~2 min), hard at 30 (~12 min).

I removed this locally only because my own operational risk changed, not because it stopped being needed — the wiring lived in src/channels/telegram.ts, which the add-telegram skill re-copies from the registry branch on every /update-nanoclaw, so the patch was silently dropped on each update.

Happy to open a PR if this is wanted — the generic watchdog module and its regression tests (bounce-that-never-completes; dead-loop-that-logs-nothing) are self-contained. One question before I do: the wiring belongs on the channels branch (src/channels/telegram.ts) but the watchdog module itself is channel-agnostic host infra and would seem to belong on main — which split would you prefer, and would you rather see this fixed in @chat-adapter/telegram upstream instead?