#15666 [BUG] AOF rewrite child remains blocked in jemalloc mutex after fork (Redis 7.4.7)

Describe the bug

A background AOF rewrite child process remained blocked for approximately 7 days and 22 hours on a production Redis 7.4.7 instance.

Redis continued serving requests, but the AOF rewrite never completed. The child process was blocked in a jemalloc mutex:

PID     PPID     STAT  WCHAN
426998  2096105  S     futex_wait_queue

Redis persistence state:

aof_rewrite_in_progress:1
aof_rewrite_scheduled:0
aof_current_rewrite_time_sec:686958
aof_last_bgrewrite_status:ok
aof_current_size:45757506678
aof_base_size:2226314945

The GDB backtrace was:

#0  __lll_lock_wait () from /lib64/libc.so.6
#1  pthread_mutex_lock@@GLIBC_2.2.5 () from /lib64/libc.so.6
#2  malloc_mutex_lock_final () at include/jemalloc/internal/mutex.h:151
#3  je_malloc_mutex_lock_slow () at src/mutex.c:90
#4  malloc_mutex_lock ()
#5  je_extent_record () at src/extent.c:921
#6  je_ecache_dalloc () at src/extent.c:147
#7  pac_dalloc_impl () at src/pac.c:277
#8  je_arena_slab_dalloc () at src/arena.c:570
#9  tcache_bin_flush_impl () at src/tcache.c:477
#10 tcache_bin_flush_bottom ()
#11 je_tcache_bin_flush_small ()
#12 tcache_gc_small ()
#13 tcache_event ()
#14 je_tcache_gc_event_handler ()
#15 je_te_event_trigger ()
#16 te_event_advance ()
#17 thread_alloc_event ()
#18 imalloc_body () at src/jemalloc.c:2585
#19 imalloc ()
#20 je_malloc_default ()
#21 ztrymalloc_usable_internal () at zmalloc.c:96
#22 zmalloc () at zmalloc.c:125
#23 rdbSaveLzfStringObject () at rdb.c:363
#24 rdbSaveRawString () at rdb.c:450
#25 rdbSaveObject () at rdb.c:855
#26 rdbSaveKeyValuePair () at rdb.c:1223
#27 rdbSaveDb () at rdb.c:1416
#28 rdbSaveRio () at rdb.c:1471
#29 rewriteAppendOnlyFile () at aof.c:2381
#30 rewriteAppendOnlyFileBackground () at aof.c:2480

This appears consistent with the forked child waiting for an allocator mutex. However, I cannot confirm whether the root cause is Redis, jemalloc, or their interaction.

Environment:

Redis: 7.4.7
OS: TencentOS
Kernel: 6.6.117-45.7.3.tl4.x86_64
Architecture: x86_64
Allocator: jemalloc-5.3.0
jemalloc-bg-thread: yes
io-threads: 1
Redis modules: none

Relevant configuration:

appendonly yes
appendfsync everysec
no-appendfsync-on-rewrite no
auto-aof-rewrite-percentage 100
auto-aof-rewrite-min-size 5368709120
aof-rewrite-incremental-fsync yes

To reproduce

The issue is rare and cannot currently be reproduced on demand.

It occurred after long-term production operation during an automatic AOF rewrite. The AOF files were approximately:

Base RDB:       2.1 GB
Incremental AOF: 3.0 GB
Current AOF:   approximately 40 GB

The rewrite child stopped making progress. Its I/O counters were:

read_bytes: 0
write_bytes: 163840

There was sufficient disk space, no swap usage, no OOM event, no filesystem error, and no apparent disk saturation.

Expected behavior

The background AOF rewrite should either complete or exit with an error so Redis can report the failure and retry it.

The rewrite child should not remain blocked indefinitely while aof_rewrite_in_progress remains set to 1.

Additional information

Sending SIGTERM did not terminate the child:

kill -TERM 426998

SIGKILL was required:

kill -KILL 426998

After the child was killed, Redis spawned a new AOF rewrite child, and the new rewrite completed within a few seconds. No data loss was observed.

The old child process had the following open files:

/data/redis/temp-rewriteaof-426998.aof
/data/redis/appendonly6500/append6500.aof.27.incr.aof
/data/redis/appendonly6500/append6500.aof.26.incr.aof

Related references:

Questions:

Does this backtrace match a known Redis or jemalloc fork-related deadlock?
Is Redis 7.4.7 with jemalloc 5.3.0 affected by a known issue?
Would upgrading Redis or jemalloc address this problem?
Is jemalloc-bg-thread yes known to interact with AOF rewrite children?
Is disabling jemalloc-bg-thread a valid mitigation?