moderator: addImagesToBlocklist silently drops ClickHouse inserts — the moderation action succeeds and the blocklist row does not exist

Surfaced while verifying the sysRedis fix for #4436 — unrelated to it, and pre-existing rather than introduced (it appears on both the pre-fix and post-fix pod generations).

addImagesToBlocklist fails intermittently when inserting to ClickHouse, and the failure is swallowed by the bestEffort wrapper:

{"type":"error","event":"image moderation side effect failed","step":"blocklist",
 "imageId":<id>,"message":"Socket hang up after 3 retries",
 "stack":"... at ClickHouseClient.insert ... at async addImagesToBlocklist
          ... at async bestEffort ... at async Promise.all (index 0)"}

The bestEffort wrapper is correct and should stay — image-moderation-effects.ts documents exactly why it exists: these run after the Image row is blocked or accepted, none of them can undo it, and when one threw "the whole action rejected and the moderator got a 500 page for work that had in fact succeeded" (reported 2026-08-12). Retrying was explicitly called the worst move available.

The problem is what the swallow leaves behind. The moderation action succeeds and the blocklist row silently does not exist. There is no retry, no queue and no reconciliation — the only trace is this log line. So the ClickHouse blocklist drifts from the moderation decisions it is supposed to record, by an amount nobody is measuring.

What I did and did not establish

• Established: the error is real, it is a ClickHouse client socket failure (Socket hang up after 3 retries, i.e. the client's own retry budget exhausted), and it reaches bestEffort so the moderator sees success.
• Established: it is not caused by the #4436 fix — the same Socket hang up class appears on the pre-fix pods.
• NOT established: the rate, or how much blocklist drift has accumulated. I saw single instances in a short window on a low-traffic evening. Do not size this from my observation — it needs a proper count over a real window before anyone decides how much it matters.
• NOT established: whether the failure is ClickHouse-side (load, restart, connection limits) or client-side (pool, timeout, keepalive).

Suggested first step

Count it before fixing it. A single Loki query over a week, grouped by step, will say whether this is a handful of rows or a systematic gap — and the step grouping is worth keeping, because image moderation side effect failed covers several different side effects with different consequences and they should not be aggregated.

If the rate turns out to be material, the fix is probably not "stop swallowing" (that reintroduces the 500 the wrapper exists to prevent) but a durable retry — the failed insert queued and replayed, rather than logged and lost.

Closing condition

Either:

• (a) a measured rate is posted in this issue over a window of at least 7 days, grouped by step, and either a PR adding durable retry/reconciliation for the blocklist step is merged, or a comment here records that the measured rate is acceptable and this closes as won't-fix; or
• (b) the underlying ClickHouse connection failure is fixed such that the same 7-day query returns zero for step: blocklist.

Checked by: @ZacxDev, by running that query and reading the result posted here.