searchd grows unbounded until OOM when disk chunk merges fall behind ingestion

Author: cfx-damirCreated Aug 24, 2026Updated Sep 15, 2026

Labelsbugwaiting

On a manticore instance with 61 GB memory and ~32 RT tables, we semi-regularly experience OOM crashes. There are times, where manticore performs perfectly fine, but at other times, it repeatedly starts to linearly grow its memory usage at a rate of ~900 MiB/min until it is OOM-killed:

During those times, manticore keeps accepting and processing queries without any particular issue (might just be slightly slower) until it crashes.

Potential cause / related

We have identified some irregularities which might be related to the issue:

Our system is set up to split tables per (logical) region with always 2 tables per region: r[index]_products and r[index]_products_aggregate. Some of these columns are MULTI64 attributes which are mostly stored using columnar storage. One of those is not using columnar storage and in one region has ~239 values per row while other regions usually have less than 15.

During indexing times (we regularly have huge amounts of ingest) the table's disk chunk merges take very long and start to back up, increasing the amount of disk chunks from the desired 16 to > ~50.

Environment

Manticore config: threads=8 optimize_cutoff=16 parallel_chunk_merges=2 rt_mem_limit=128M

Host container in AWS: 61 GB memory limit, 8 vCPU, local NVMe for storage.

Logs

Logs when memory starts to grow unbounded:

23:54 rt: table r13_products_aggregate: diskchunk 59796(17), segments 25  saved in 0.464710 (0.464919) sec [attrs=144ms deadmap=0ms docs=237ms infix=62ms checkpoints=0ms header=0ms], RAM saved/new 45398333/3838060 ratio 0.922048 (soft limit 123755229, conf limit 134217728)
23:54 rt: table r15_products_aggregate: diskchunk 17(18), segments 25  saved in 0.672777 (0.673027) sec [attrs=429ms deadmap=0ms docs=173ms infix=49ms checkpoints=0ms header=0ms], RAM saved/new 45657255/2601410 ratio 0.946094 (soft limit 126982647, conf limit 134217728)
23:54 rt: table r15_products_aggregate: merged chunks /var/lib/manticore/r15_products_aggregate/r15_products_aggregate.16, /var/lib/manticore/r15_products_aggregate/r15_products_aggregate.17 to /var/lib/manticore/r15_products_aggregate/r15_products_aggregate.18 in 1s (build 1s: attrs 821ms, words 327ms, finalize 84ms; publish 3ms). Remaining chunk count: 17
23:54 rt: table r15_products: diskchunk 19(20), segments 27  saved in 0.812638 (0.812846) sec [attrs=515ms deadmap=0ms docs=215ms infix=62ms checkpoints=0ms header=0ms], RAM saved/new 45903058/829906 ratio 0.950000 (soft limit 127506841, conf limit 134217728)
23:54 rt: table r15_products: merged chunks /var/lib/manticore/r15_products/r15_products.18, /var/lib/manticore/r15_products/r15_products.19 to /var/lib/manticore/r15_products/r15_products.20 in 2s (build 2s: attrs 1s, words 524ms, finalize 142ms; publish 3ms). Remaining chunk count: 19
23:54 rt: table r13_products: diskchunk 34114(17), segments 32  saved in 0.612090 (0.618221) sec [attrs=167ms deadmap=0ms docs=334ms infix=90ms checkpoints=0ms header=0ms], RAM saved/new 44645787/0 ratio 0.950000 (soft limit 127506841, conf limit 134217728)
23:54 rt: table r13_products_aggregate: merged chunks /var/lib/manticore/r13_products_aggregate/r13_products_aggregate.59749, /var/lib/manticore/r13_products_aggregate/r13_products_aggregate.59796 to /var/lib/manticore/r13_products_aggregate/r13_products_aggregate.59797 in 47s (build 47s: attrs 20s, words 25s, finalize 2s; publish 45ms). Remaining chunk count: 16
23:54 rt: table r13_products_aggregate: optimized progressive chunk(s) 17 ( left 16 ) in 46s 947ms
23:54 rt: table r13_products_aggregate: diskchunk 59798(17), segments 23  saved in 0.659757 (0.659952) sec [attrs=214ms deadmap=0ms docs=340ms infix=84ms checkpoints=0ms header=0ms], RAM saved/new 65688828/4673558 ratio 0.933579 (soft limit 125302818, conf limit 134217728)
23:55 rt: table r15_products_aggregate: diskchunk 19(18), segments 26  saved in 1.843844 (1.844058) sec [attrs=1234ms deadmap=0ms docs=439ms infix=101ms checkpoints=0ms header=0ms], RAM saved/new 127797078/3618556 ratio 0.950000 (soft limit 127506841, conf limit 134217728)
23:55 rt: table r15_products_aggregate: optimized progressive chunk(s) 18 ( left 18 ) in 95ms 126us
23:55 rt: table r13_products: merged chunks /var/lib/manticore/r13_products/r13_products.34075, /var/lib/manticore/r13_products/r13_products.34114 to /var/lib/manticore/r13_products/r13_products.34115 in 38s (build 38s: attrs 16s, words 21s, finalize 2s; publish 28ms). Remaining chunk count: 16
23:55 rt: table r13_products: optimized progressive chunk(s) 17 ( left 16 ) in 38s 549ms
23:55 rt: table r15_products: merged chunks /var/lib/manticore/r15_products/r15_products.0, /var/lib/manticore/r15_products/r15_products.20 to /var/lib/manticore/r15_products/r15_products.21 in 59s (build 59s: attrs 44s, words 13s, finalize 1s; publish 23ms). Remaining chunk count: 18
23:55 rt: table r15_products: diskchunk 22(19), segments 27  saved in 2.083083 (2.083271) sec [attrs=1419ms deadmap=0ms docs=516ms infix=127ms checkpoints=0ms header=0ms], RAM saved/new 128316190/3488333 ratio 0.950000 (soft limit 127506841, conf limit 134217728)
23:55 rt: table r13_products: diskchunk 34116(17), segments 23  saved in 0.682206 (0.688234) sec [attrs=194ms deadmap=0ms docs=374ms infix=103ms checkpoints=0ms header=0ms], RAM saved/new 53348018/0 ratio 0.950000 (soft limit 127506841, conf limit 134217728)
23:55 rt: table r13_products_aggregate: merged chunks /var/lib/manticore/r13_products_aggregate/r13_products_aggregate.59783, /var/lib/manticore/r13_products_aggregate/r13_products_aggregate.59798 to /var/lib/manticore/r13_products_aggregate/r13_products_aggregate.59799 in 47s (build 47s: attrs 20s, words 26s, finalize 2s; publish 38ms). Remaining chunk count: 16
23:55 rt: table r13_products_aggregate: optimized progressive chunk(s) 17 ( left 16 ) in 47s 106ms
23:55 rt: table r15_products_aggregate: diskchunk 20(19), segments 27  saved in 1.855559 (1.888449) sec [attrs=1307ms deadmap=0ms docs=431ms infix=98ms checkpoints=0ms header=0ms], RAM saved/new 128474149/3336808 ratio 0.950000 (soft limit 127506841, conf limit 134217728)
23:55 rt: table r15_products_aggregate: merged chunks /var/lib/manticore/r15_products_aggregate/r15_products_aggregate.12, /var/lib/manticore/r15_products_aggregate/r15_products_aggregate.18 to /var/lib/manticore/r15_products_aggregate/r15_products_aggregate.21 in 2m (build 2m: attrs 1m, words 25s, finalize 1s; publish 50ms). Remaining chunk count: 18
23:55 rt: table r15_products_aggregate: optimized progressive chunk(s) 20 ( left 18 ) in 1m 43.6s
23:55 rt: table r13_products_aggregate: diskchunk 59800(17), segments 24  saved in 0.602662 (0.622226) sec [attrs=192ms deadmap=0ms docs=314ms infix=78ms checkpoints=0ms header=0ms], RAM saved/new 62644230/0 ratio 0.950000 (soft limit 127506841, conf limit 134217728)
23:55 rt: table r15_products_aggregate: merged chunks /var/lib/manticore/r15_products_aggregate/r15_products_aggregate.19, /var/lib/manticore/r15_products_aggregate/r15_products_aggregate.20 to /var/lib/manticore/r15_products_aggregate/r15_products_aggregate.22 in 4s (build 4s: attrs 3s, words 952ms, finalize 173ms; publish 4ms). Remaining chunk count: 17
23:56 rt: table r13_products: merged chunks /var/lib/manticore/r13_products/r13_products.34091, /var/lib/manticore/r13_products/r13_products.34116 to /var/lib/manticore/r13_products/r13_products.34117 in 38s (build 38s: attrs 16s, words 20s, finalize 2s; publish 32ms). Remaining chunk count: 16
23:56 rt: table r13_products: optimized progressive chunk(s) 17 ( left 16 ) in 38s 286ms
23:56 rt: table r13_products_aggregate: merged chunks /var/lib/manticore/r13_products_aggregate/r13_products_aggregate.59765, /var/lib/manticore/r13_products_aggregate/r13_products_aggregate.59800 to /var/lib/manticore/r13_products_aggregate/r13_products_aggregate.59801 in 46s (build 46s: attrs 19s, words 25s, finalize 2s; publish 39ms). Remaining chunk count: 16
23:56 rt: table r13_products_aggregate: optimized progressive chunk(s) 17 ( left 16 ) in 46s 761ms
23:56 rt: table r15_products_aggregate: diskchunk 23(18), segments 27  saved in 1.841249 (1.841457) sec [attrs=1307ms deadmap=0ms docs=421ms infix=95ms checkpoints=0ms header=0ms], RAM saved/new 128527252/4286757 ratio 0.950000 (soft limit 127506841, conf limit 134217728)
23:56 rt: table r15_products_aggregate: optimized progressive chunk(s) 18 ( left 18 ) in 89ms 478us
23:56 rt: table r15_products: merged chunks /var/lib/manticore/r15_products/r15_products.3, /var/lib/manticore/r15_products/r15_products.22 to /var/lib/manticore/r15_products/r15_products.24 in 1m (build 1m: attrs 53s, words 15s, finalize 1s; publish 24ms). Remaining chunk count: 18
23:56 rt: table r15_products: optimized progressive chunk(s) 19 ( left 18 ) in 1m 9.2s
23:56 rt: table r15_products: diskchunk 23(19), segments 27  saved in 2.088451 (2.119104) sec [attrs=1442ms deadmap=0ms docs=508ms infix=125ms checkpoints=0ms header=0ms], RAM saved/new 127694163/5210018 ratio 0.950000 (soft limit 127506841, conf limit 134217728)
23:57 rt: table r13_products_aggregate: diskchunk 59802(17), segments 23  saved in 0.581080 (0.587695) sec [attrs=184ms deadmap=0ms docs=305ms infix=74ms checkpoints=0ms header=0ms], RAM saved/new 60092630/0 ratio 0.950000 (soft limit 127506841, conf limit 134217728)
23:57 rt: table r15_products_aggregate: diskchunk 24(19), segments 26  saved in 1.791671 (1.791889) sec [attrs=1247ms deadmap=0ms docs=426ms infix=98ms checkpoints=0ms header=0ms], RAM saved/new 128131033/2522169 ratio 0.950000 (soft limit 127506841, conf limit 134217728)
23:57 rt: table r15_products_aggregate: merged chunks /var/lib/manticore/r15_products_aggregate/r15_products_aggregate.23, /var/lib/manticore/r15_products_aggregate/r15_products_aggregate.24 to /var/lib/manticore/r15_products_aggregate/r15_products_aggregate.25 in 4s (build 4s: attrs 3s, words 946ms, finalize 173ms; publish 4ms). Remaining chunk count: 18
23:57 rt: table r15_products_aggregate: optimized progressive chunk(s) 19 ( left 18 ) in 4s 57ms
23:57 rt: table r13_products: diskchunk 34118(17), segments 24  saved in 0.685488 (0.691590) sec [attrs=197ms deadmap=0ms docs=375ms infix=102ms checkpoints=0ms header=0ms], RAM saved/new 54963926/0 ratio 0.950000 (soft limit 127506841, conf limit 134217728)
23:57 rt: table r15_products: merged chunks /var/lib/manticore/r15_products/r15_products.2, /var/lib/manticore/r15_products/r15_products.23 to /var/lib/manticore/r15_products/r15_products.25 in 1m (build 1m: attrs 48s, words 14s, finalize 1s; publish 328ms). Remaining chunk count: 18
23:57 rt: table r15_products: optimized progressive chunk(s) 19 ( left 18 ) in 1m 3.6s
23:57 rt: table r15_products_aggregate: merged chunks /var/lib/manticore/r15_products_aggregate/r15_products_aggregate.0, /var/lib/manticore/r15_products_aggregate/r15_products_aggregate.22 to /var/lib/manticore/r15_products_aggregate/r15_products_aggregate.26 in 2m (build 2m: attrs 1m, words 26s, finalize 1s; publish 734ms). Remaining chunk count: 17
23:57 rt: table r15_products: diskchunk 26(19), segments 26  saved in 2.078048 (2.078242) sec [attrs=1428ms deadmap=0ms docs=504ms infix=124ms checkpoints=0ms header=0ms], RAM saved/new 127545992/5790442 ratio 0.950000 (soft limit 127506841, conf limit 134217728)
23:57 rt: table r13_products_aggregate: merged chunks /var/lib/manticore/r13_products_aggregate/r13_products_aggregate.59769, /var/lib/manticore/r13_products_aggregate/r13_products_aggregate.59802 to /var/lib/manticore/r13_products_aggregate/r13_products_aggregate.59803 in 46s (build 46s: attrs 20s, words 25s, finalize 2s; publish 41ms). Remaining chunk count: 16
23:57 rt: table r13_products_aggregate: optimized progressive chunk(s) 17 ( left 16 ) in 46s 526ms
23:58 rt: table r13_products: merged chunks /var/lib/manticore/r13_products/r13_products.34095, /var/lib/manticore/r13_products/r13_products.34118 to /var/lib/manticore/r13_products/r13_products.34119 in 38s (build 38s: attrs 16s, words 21s, finalize 2s; publish 33ms). Remaining chunk count: 16
23:58 rt: table r13_products: optimized progressive chunk(s) 17 ( left 16 ) in 37s 964ms
23:58 rt: table r15_products_aggregate: diskchunk 27(18), segments 27  saved in 1.819503 (1.849126) sec [attrs=1282ms deadmap=0ms docs=418ms infix=97ms checkpoints=0ms header=0ms], RAM saved/new 128455226/1954235 ratio 0.950000 (soft limit 127506841, conf limit 134217728)
23:58 rt: table r15_products_aggregate: optimized progressive chunk(s) 18 ( left 18 ) in 84ms 187us
23:58 rt: table r13_products_aggregate: diskchunk 59804(17), segments 23  saved in 0.577426 (0.584287) sec [attrs=187ms deadmap=0ms docs=298ms infix=74ms checkpoints=0ms header=0ms], RAM saved/new 62233559/0 ratio 0.950000 (soft limit 127506841, conf limit 134217728)
23:58 rt: table r15_products: merged chunks /var/lib/manticore/r15_products/r15_products.4, /var/lib/manticore/r15_products/r15_products.26 to /var/lib/manticore/r15_products/r15_products.27 in 1m (build 1m: attrs 49s, words 15s, finalize 1s; publish 27ms). Remaining chunk count: 18
23:58 rt: table r15_products: optimized progressive chunk(s) 19 ( left 18 ) in 1m 5s

Logs when manticore crashes:


rt: table r13_products_aggregate: merged chunks /var/lib/manticore/r13_products_aggregate/r13_products_aggregate.59851, /var/lib/manticore/r13_products_aggregate/r13_products_aggregate.59890 to /var/lib/manticore/r13_products_aggregate/r13_products_aggregate.59891 in 47s (build 46s: attrs 20s, words 25s, finalize 2s; publish 376ms). Remaining chunk count: 16
rt: table r13_products_aggregate: optimized progressive chunk(s) 17 ( left 16 ) in 47s 164ms
rt: table r13_products: diskchunk 34176(17), segments 24  saved in 0.788518 (0.792793) sec [attrs=217ms deadmap=0ms docs=438ms infix=110ms checkpoints=0ms header=0ms], RAM saved/new 62306049/0 ratio 0.950000 (soft limit 127506841, conf limit 134217728)
rt: table r13_products_aggregate: diskchunk 59892(17), segments 23  saved in 0.648273 (0.655326) sec [attrs=202ms deadmap=0ms docs=344ms infix=81ms checkpoints=0ms header=0ms], RAM saved/new 69880135/0 ratio 0.950000 (soft limit 127506841, conf limit 134217728)
rt: table r15_products_aggregate: merged chunks /var/lib/manticore/r15_products_aggregate/r15_products_aggregate.50, /var/lib/manticore/r15_products_aggregate/r15_products_aggregate.146 to /var/lib/manticore/r15_products_aggregate/r15_products_aggregate.147 in 2m (build 2m: attrs 1m, words 26s, finalize 1s; publish 1s). Remaining chunk count: 16
rt: table r15_products_aggregate: optimized progressive chunk(s) 17 ( left 16 ) in 1m 49.8s
rt: table r13_products: merged chunks /var/lib/manticore/r13_products/r13_products.34141, /var/lib/manticore/r13_products/r13_products.34176 to /var/lib/manticore/r13_products/r13_products.34177 in 39s (build 39s: attrs 16s, words 21s, finalize 2s; publish 414ms). Remaining chunk count: 16
rt: table r13_products: optimized progressive chunk(s) 17 ( left 16 ) in 39s 249ms
rt: table r13_products_aggregate: merged chunks /var/lib/manticore/r13_products_aggregate/r13_products_aggregate.59863, /var/lib/manticore/r13_products_aggregate/r13_products_aggregate.59892 to /var/lib/manticore/r13_products_aggregate/r13_products_aggregate.59893 in 47s (build 46s: attrs 19s, words 25s, finalize 2s; publish 558ms). Remaining chunk count: 16
rt: table r13_products_aggregate: optimized progressive chunk(s) 17 ( left 16 ) in 47s 171ms
rt: table r13_products_aggregate: diskchunk 59894(17), segments 23  saved in 0.654265 (0.662682) sec [attrs=206ms deadmap=0ms docs=348ms infix=78ms checkpoints=0ms header=0ms], RAM saved/new 72205175/0 ratio 0.950000 (soft limit 127506841, conf limit 134217728)
rt: table r13_products: diskchunk 34178(17), segments 24  saved in 0.791741 (0.798218) sec [attrs=218ms deadmap=0ms docs=439ms infix=110ms checkpoints=0ms header=0ms], RAM saved/new 64095895/0 ratio 0.950000 (soft limit 127506841, conf limit 134217728)
rt: table r13_products_aggregate: merged chunks /var/lib/manticore/r13_products_aggregate/r13_products_aggregate.59865, /var/lib/manticore/r13_products_aggregate/r13_products_aggregate.59894 to /var/lib/manticore/r13_products_aggregate/r13_products_aggregate.59895 in 47s (build 46s: attrs 19s, words 25s, finalize 2s; publish 568ms). Remaining chunk count: 16
rt: table r13_products_aggregate: optimized progressive chunk(s) 17 ( left 16 ) in 46s 881ms
rt: table r13_products_aggregate: diskchunk 59896(17), segments 23  saved in 0.635628 (0.641353) sec [attrs=203ms deadmap=0ms docs=343ms infix=79ms checkpoints=0ms header=0ms], RAM saved/new 68664293/0 ratio 0.950000 (soft limit 127506841, conf limit 134217728)
rt: table r13_products: merged chunks /var/lib/manticore/r13_products/r13_products.34159, /var/lib/manticore/r13_products/r13_products.34178 to /var/lib/manticore/r13_products/r13_products.34179 in 37s (build 37s: attrs 15s, words 20s, finalize 2s; publish 40ms). Remaining chunk count: 16
rt: table r13_products: optimized progressive chunk(s) 17 ( left 16 ) in 37s 790ms
FATAL: out of memory (unable to allocate 34359738368 bytes)

Side question: is parallel_chunk_merges = max(1, min(2, threads/2)) intentional in that it stops scaling at 4 threads? On a 16-core box it still yields 2, so more CPUs give no extra merge capacity unless set explicitly.

Source: manticoresoftware/manticoresearch

View original on GitHub View discussion on GitHub