[BUG] Redis keys for block_run accumulate indefinitely due to missing TTL
Author: baydakovssCreated May 5, 2026Updated Jul 8, 2026
Labelsbug
Mage version
0.9.76
Describe the bug
When using Redis-backed ProcessQueue, job keys are written to Redis without an expiration time:
# mage_ai/orchestration/queue/process_queue.py:116
self.redis_client.set(job_id, self.client_id) # no TTLOver time this causes unbounded growth of keys in Redis, leading to increased memory usage.
Actual behavior Job keys never expire. TTL returns -1 for all job_id keys.
valkey-cli -h rw.mageai.cache.prod.xxx.yyy -n 15
rw.mageai.cache.prod.xxx.yyy:6379[15]> scan 0 count 1
1) "1048576"
2) 1) "block_run_658120"
2) "block_run_1372533"
3) "block_run_3846032"
rw.mageai.cache.prod.xxx.team:6379[15]> TTL "block_run_658120"
(integer) -1
rw.mageai.cache.prod.xxx.yyy:6379[15]> DBSIZE
(integer) 4211665To reproduce
Steps to reproduce
- Configure Mage with REDIS_URL
- Run pipelines over time
- Check Redis:
redis-cli --scan | wc -l- key count grows indefinitely
Expected behavior
Job keys should expire after a reasonable timeout (e.g. 24 hours), similar to other keys in the same file which use ex=LIVENESS_TIMEOUT_SECONDS.
Screenshots
No response
Operating system
No response
Additional context
No response
Source: mage-ai/mage-ai