Feature request: inspect and manually redeliver failed terminal webhooks
User problem
Firecrawl's crawl and batch endpoints are often the first step in a customer's ingestion pipeline: scrape → webhook → transform/index. If the customer's receiver is down during the terminal webhook, the Firecrawl job may have succeeded but the downstream pipeline never starts. An automatic retry window helps with transient outages; operators still need a controlled way to recover one known failed terminal delivery after their receiver is fixed, without paying for and re-running the crawl.
The open-source API currently records webhook success/failure in webhook_logs (delivery, log reads), but those logs do not retain a replayable payload or expose a redelivery operation. The existing RabbitMQ queue (#2534) and the open consumer/retry/DLQ PR #3182 address automatic delivery reliability; this proposal is the operator recovery step after retries are exhausted.
Proposed module: terminal webhook delivery history + manual redelivery
Scope the first version to the terminal events defined in the current WebhookEvent enum: crawl.completed, batch_scrape.completed, extract.completed, and extract.failed. The latter carries a failed extraction outcome; redelivery is only for a failed HTTP delivery, not a rerun of extraction. Page-level events can remain out of scope.
- Keep a short-lived, access-controlled delivery record with team/job/event, original
webhookId, attempt count, status, timestamp, destination identity, and the exact payload needed for redelivery. Do not store authorization headers or webhook secrets in that record. Apply an explicit retention limit. - Let the owning team inspect failed terminal deliveries and request redelivery of one delivery ID through a rate-limited API. Only send to the original webhook destination/configuration; do not accept an arbitrary URL from the replay request. Re-run the existing SSRF/destination checks and sign the new HTTP attempt with the current configured secret.
- Preserve the logical event's
webhookIdacross attempts so receivers can deduplicate, while assigning a separate attempt ID for audit. Mark the attempt as a redelivery and expose its outcome. - Reject redelivery for another team, an expired record, a still-in-flight attempt, or a delivery already marked successful unless an explicit safe policy is defined.
This can be optional for self-hosted deployments where retention and storage cost matter. It does not require re-scraping pages or building a general job scheduler.
Acceptance criteria
- Simulate a completed crawl whose receiver returns 503; after the receiver recovers, redeliver that terminal event without starting a new crawl or charging it again.
- The receiver gets the original event identity and payload; each attempt has its own recorded status.
- Unauthorized, expired, duplicate concurrent, and arbitrary-destination redelivery attempts are rejected.
- Secrets are absent from delivery records and logs; signing and SSRF protections still apply.
- Tests cover both inline delivery and the RabbitMQ path when enabled.
I searched current code, docs, issues, and PRs for webhook retry, replay, redelivery, and dead-letter handling. I found automatic retry/DLQ work in #3182, but no existing user-facing terminal-event redelivery module.
Source: firecrawl/firecrawl