[BUG] Bulk Inbox/Account deletion starves live WhatsApp/inbound webhooks due to Sidekiq queue priority inversion

Description

When an administrator deletes an Inbox (or Account) with a large number of conversations (e.g., 5,000–10,000+), incoming live WhatsApp customer messages (Webhooks::WhatsappEventsJob) and other webhooks assigned to :low are completely starved and blocked from processing for 30–60+ minutes.

Live customer communication is halted because background deletion jobs monopolize worker threads.

---

Root Cause Analysis

1. Queue Assignment Inconsistency:

   • In app/jobs/webhooks/whatsapp_events_job.rb, inbound webhooks are queued as :low:
     ruby复制

     class Webhooks::WhatsappEventsJob < MutexApplicationJob
       queue_as :low

   • In contrast, other channel webhook handlers (e.g., facebook_events_job.rb, instagram_events_job.rb, telegram_events_job.rb) use queue_as :default.
   • The deletion initiator DeleteObjectJob is also queued as queue_as :low.

2. Async Cascade Fan-Out into Higher-Priority Queues:

   • When DeleteObjectJob runs, purge_heavy_associations invokes batch.each(&:destroy!) on all conversations in the inbox.
   • Each Conversation#destroy! triggers Rails ActiveRecord asynchronous associations:
     • has_many :messages, dependent: :destroy_async -> enqueues ActiveRecord::DestroyAssociationAsyncJob into :default.
     • Associated attachments -> enqueue ActiveStorage::PurgeJob into :default.
     • EventDispatcherJob (conversation.deleted) -> enqueues into :critical.
   • An inbox with ~7,500 conversations generates 10,000+ jobs in :default and hundreds in :critical.

3. Strict Sidekiq Queue Priority Ordering:

   • In config/sidekiq.yml, queues are defined without weights:
     yaml复制

     :queues:
       - critical
       - high
       - medium
       - default
       - mailers
       - action_mailbox_routing
       - low

   • In Sidekiq, unweighted queues enforce strict top-to-bottom priority.
   • Sidekiq worker threads will never pull a single job from :low as long as :default has jobs waiting.
   • As a result, all incoming WhatsApp webhooks arriving in :low sit backlogged for 30–60+ minutes until every single deleted message and attachment is cleaned up.

---

Steps to Reproduce

1. Have an inbox with ~5,000 to 10,000 conversations.
2. In Chatwoot UI, delete the inbox.
3. Check Redis queue lengths (redis-cli llen queue:default) — notice it immediately jumps to 10,000+ jobs.
4. Send an incoming WhatsApp message to another active inbox.
5. The message webhook is accepted by the Rails web server (200 OK) and enqueued to queue:low.
6. Inspect Sidekiq worker: all worker threads are 100% occupied processing queue:default (ActiveRecord::DestroyAssociationAsyncJob).
7. The inbound WhatsApp message does not appear in the dashboard until all deletion jobs in queue:default have finished.

---

Environment

• Chatwoot version: v4.x / latest (develop)
• Sidekiq: 7.3.x
• OS: Linux (Ubuntu 22.04 LTS)

---

Proposed Solutions

1. Prioritize Real-Time Inbound Webhooks: Inbound customer communication (Webhooks::WhatsappEventsJob, Webhooks::TwilioEventsJob) is real-time and business-critical. It should be assigned to :high (matching SendReplyJob) or at least :default to be consistent with FacebookEventsJob and TelegramEventsJob.

2. Route Bulk Deletion Cascades to Lower/Housekeeping Queues: config/sidekiq.yml already defines queues such as purgable and housekeeping at the bottom. Bulk deletion jobs (ActiveRecord::DestroyAssociationAsyncJob and ActiveStorage::PurgeJob triggered during bulk object deletion) should not pollute :default.

3. Consider Queue Weights in Default Sidekiq Config: Adding weights in config/sidekiq.yml (e.g., ['high', 5], ['default', 3], ['low', 2]) prevents total starvation of lower queues during high-volume spikes in higher queues.