#6102·mage-ai

[BUG] Redis keys for block_run accumulate indefinitely due to missing TTL

Author: baydakovssCreated May 5, 2026Updated Jul 8, 2026
Labelsbug

Mage version

0.9.76

Describe the bug

When using Redis-backed ProcessQueue, job keys are written to Redis without an expiration time:

python
# mage_ai/orchestration/queue/process_queue.py:116
self.redis_client.set(job_id, self.client_id)  # no TTL

Over time this causes unbounded growth of keys in Redis, leading to increased memory usage.

Actual behavior Job keys never expire. TTL returns -1 for all job_id keys.

valkey-cli -h rw.mageai.cache.prod.xxx.yyy -n 15
rw.mageai.cache.prod.xxx.yyy:6379[15]> scan 0 count 1
1) "1048576"
2) 1) "block_run_658120"
   2) "block_run_1372533"
   3) "block_run_3846032"
rw.mageai.cache.prod.xxx.team:6379[15]> TTL "block_run_658120"
(integer) -1
rw.mageai.cache.prod.xxx.yyy:6379[15]> DBSIZE
(integer) 4211665

To reproduce

Steps to reproduce

  1. Configure Mage with REDIS_URL
  2. Run pipelines over time
  3. Check Redis: redis-cli --scan | wc -l - key count grows indefinitely

Expected behavior

Job keys should expire after a reasonable timeout (e.g. 24 hours), similar to other keys in the same file which use ex=LIVENESS_TIMEOUT_SECONDS.

Screenshots

No response

Operating system

No response

Additional context

No response