worker.lua:164: failed to store event: queue overflow causes mlcache invalidation loss and stale upstream targets

Is there an existing issue for this?

• I have searched the existing issues (related: #13452, #11103)

Kong version

3.0.1 (also reproduced in 3.2.2 per #13452)

Current Behavior

Under high traffic with frequent upstream target changes, Kong logs:

[error] 63#0: *3 [lua] worker.lua:164: failed to store event: queue overflow.
data is :{"source":"mlcache","wid":0,"data":"targets:fake-2:10.0.0.2:20002::::","event":"mlcache:invalidations:kong_core_db_cache"}, context: ngx.timer

When this happens, mlcache invalidation events for targets entities are silently dropped:

• The Kong balancer's in-memory target list is not refreshed in affected workers
• Traffic continues routing to stale/deleted targets
• If overflow is continuous, the stale state never recovers — each DB poll cycle also drops its invalidation events

Confirmed by Kong contributor @oowl in #13452:

"Due to the worker events system being broken, the Kong Balancer system can not refresh all worker data structs that contain all target and upstream statuses"

Root Cause

Event flow for mlcache invalidation:

1. mlcache:delete(key) → ev:publish("all", ...) → _pub_queue
2. write_thread → broker unix socket
3. broker → all worker sockets
4. read_thread → _sub_queue (capacity: max_queue_len = 1024*50 = 51200 in kong/global.lua)
5. events_thread → do_event callback

When events_thread (step 5) is slow under load and read_thread (step 4) fills _sub_queue faster than it is consumed, events are permanently dropped with no retry mechanism.

Reproduction (verified on Kong 3.0.1)

Three changes to reproduce locally:

1. Shrink queue (kong/global.lua):

max_queue_len = 100,  -- default 1024*50

2. Inject slow consumer (resty/events/worker.lua after do_event):

sleep(0.05)  -- simulate slow callback under load

3. Burst plugin — publish 200 invalidation events per request in access phase:

for i = 1, 200 do
  kong.worker_events.post("mlcache", "mlcache:invalidations:kong_core_db_cache",
    string.format("targets:fake-%d:10.0.0.%d:%d::::", i, i%254, 20000+i))
end

50 concurrent requests → 100+ queue overflow errors immediately.

Impact Verification

1. Route upstream to backend-A:9001
2. Trigger queue overflow
3. Switch target to backend-B:9002 via Admin API
4. Immediately: requests still return backend-A (stale cache)
5. After 35s (next DB poll): requests correctly return backend-B

With continuous overflow: cache never recovers.

Expected Behavior

Options to fix:

1. Back-pressure: slow read_thread when _sub_queue near capacity instead of dropping
2. Retry: retain dropped events and retry (similar to lua-resty-events PR #61 for send failures)
3. Expose config: make max_queue_len a Kong config parameter for operators to tune

Related

• #13452 — identical symptom (3.2.2), closed without fix
• #11103 — related IPC socket failure, closed without fix
• kong PR #13228 / lua-resty-events PR #60 #61 (Kong 3.7.0) — improve event retention on connection failures, but do not address queue overflow under sustained load