[Design] Worker & retry architecture rework (Celery / queue migration)

The theme

The current worker (surreal-commands) has architectural limits that surface as a recurring family of bugs: jobs can't be truly cancelled, retries fire for the wrong reasons, and there's no idempotency story. This issue is the home for the worker & retry architecture rework — likely a migration to a proper task queue (Celery? FastAPI background tasks? something else) — and the requirements that rework must satisfy.

Which worker engine do we move to, and what retry/cancellation/idempotency guarantees does the new architecture provide?

Requirements folded in (only the rework truly fixes these)

• Cancellation (was #886) — deleting a source leaves zombie commands that block the worker. surreal-commands can't truly cancel — it can only delete DB rows. Real cancellation needs the new engine.
• Correct retry semantics (was #892) — the current retry architecture is wrong: DB timeouts trigger LLM retries, causing spam. The rework must define what is retried and when (transient infra vs. real failure).

The questions to settle (design)

1. Engine — Celery, FastAPI background tasks, RQ, or other? Trade-offs vs. our SurrealDB-centric stack.
2. Cancellation — first-class job cancellation (covers #886).
3. Retry policy — distinguish infra timeouts from genuine failures; backoff; no LLM spam (covers #892).
4. Idempotency — see #891 (kept separate as a near-term fix, but the new architecture should make it natural).
5. Migration path — how we move off surreal-commands without breaking running instances.

Related but tracked separately

• #891 — idempotency on insight creation (a discrete near-term fix; doesn't require the new engine).
• #893 — sequential processing mode for single-GPU (already ready; independent config flag).
• #644 — worker can't load external Ollama credentials (different root cause: credential loading in worker context).

Consolidates #886, #892.