How to fine-tune settings for large memory requests, frequently larger than 8MB
Author: 13015517713Created Apr 3, 2025Updated Sep 2, 2026
env: linux 5.4 x86_64, jemalloc version 5.3
I'm having some performance issues and would like some advice.
My programs would frequently request memory with large size. They were universally larger than 15MB. By using perf, I found a lot of overhead in page faults. I wrote a simple program to test this, a single thread to loop through memory requests and memset. Like that:
while(true){
void* p = malloc(g_size);
memset(p, 0, g_size);
free(p);
}I found that when g_size=7MB there is no page fault and the qps is about 5000, then when I request 7.1MB the qps drops to 600 with a lot of page faults as described above. In the issue I learnt that the oversize_threshold could have an effect on this, so I set the oversize_threshold to 16MB. the results didn't change.
7MB PerfReport
7.1MB PerfReport
I have some questions about this.
- What jemalloc does internally for large memory
- When I set oversize_threshold (print_stat=true), the dedicated arena doesn't show up and it seems to be set successfully. But results haven't changed.
- For large memory chunks, it would be nice to fine tune the parameters so that they can be managed like small and large memory, although it may not be possible with tcache
print_stats=true, for 7MB, run about 20s
Version: "5.3.0-0-g54eaed1d8b56b1aa528be3bdd1877e59c56fa90c"
Build-time option settings
config.cache_oblivious: true
config.debug: false
config.fill: true
config.lazy_lock: true
config.malloc_conf: ""
config.opt_safety_checks: true
config.prof: true
config.prof_libgcc: true
config.prof_libunwind: false
config.stats: true
config.utrace: false
config.xmalloc: true
Run-time option settings
opt.abort: false
opt.abort_conf: false
opt.cache_oblivious: true
opt.confirm_conf: false
opt.retain: true
opt.dss: "secondary"
opt.narenas: 1
opt.percpu_arena: "disabled"
opt.oversize_threshold: 16777216
opt.hpa: false
opt.hpa_slab_max_alloc: 65536
opt.hpa_hugification_threshold: 1992294
opt.hpa_hugify_delay_ms: 10000
opt.hpa_min_purge_interval_ms: 5000
opt.hpa_dirty_mult: "0.25"
opt.hpa_sec_nshards: 4
opt.hpa_sec_max_alloc: 32768
opt.hpa_sec_max_bytes: 262144
opt.hpa_sec_bytes_after_flush: 131072
opt.hpa_sec_batch_fill_extra: 0
opt.metadata_thp: "disabled"
opt.mutex_max_spin: 600
opt.background_thread: true (background_thread: true)
opt.dirty_decay_ms: 1000 (arenas.dirty_decay_ms: 1000)
opt.muzzy_decay_ms: 1000 (arenas.muzzy_decay_ms: 1000)
opt.lg_extent_max_active_fit: 6
opt.junk: "false"
opt.zero: false
opt.xmalloc: false
opt.experimental_infallible_new: false
opt.tcache: false
opt.tcache_max: 32768
opt.tcache_nslots_small_min: 20
opt.tcache_nslots_small_max: 200
opt.tcache_nslots_large: 20
opt.lg_tcache_nslots_mul: 1
opt.tcache_gc_incr_bytes: 65536
opt.tcache_gc_delay_bytes: 0
opt.lg_tcache_flush_small_div: 1
opt.lg_tcache_flush_large_div: 1
opt.thp: "default"
opt.prof: false
opt.prof_prefix: "jeprof"
opt.prof_active: true (prof.active: false)
opt.prof_thread_active_init: true (prof.thread_active_init: false)
opt.lg_prof_sample: 19 (prof.lg_sample: 0)
opt.prof_accum: false
opt.lg_prof_interval: -1
opt.prof_gdump: false
opt.prof_final: false
opt.prof_leak: false
opt.prof_leak_error: false
opt.stats_print: true
opt.stats_print_opts: ""
opt.stats_print: true
opt.stats_print_opts: ""
opt.stats_interval: -1
opt.stats_interval_opts: ""
opt.zero_realloc: "free"
Profiling settings
prof.thread_active_init: false
prof.active: false
prof.gdump: false
prof.interval: 0
prof.lg_sample: 0
Arenas: 2
Quantum size: 16
Page size: 4096
Maximum thread-cached size class: 32768
Number of bin size classes: 36
Number of thread-cache bin size classes: 41
Number of large size classes: 196
Allocated: 7488824, active: 7532544, metadata: 2441040 (n_thp 0), resident: 9912320, mapped: 13824000, retained: 2953216
Count of realloc(non-null-ptr, 0) calls: 0
Background threads: 1, num_runs: 1343, run_interval: 34971521 ns
n_lock_ops (#/sec) n_waiting (#/sec) n_spin_acq (#/sec) n_owner_switch (#/sec) total_wait_ns (#/sec) max_wait_ns max_n_thds
background_thread 5 0 0 0 0 0 1 0 0 0 0 0
max_per_bg_thd 2201 46 0 0 0 0 4 0 0 0 0 0
ctl 2 0 0 0 0 0 1 0 0 0 0 0
prof 0 0 0 0 0 0 0 0 0 0 0 0
prof_thds_data 0 0 0 0 0 0 0 0 0 0 0 0
prof_dump 0 0 0 0 0 0 0 0 0 0 0 0
prof_recent_alloc 0 0 0 0 0 0 0 0 0 0 0 0
prof_recent_dump 0 0 0 0 0 0 0 0 0 0 0 0
prof_stats 0 0 0 0 0 0 0 0 0 0 0 0
arenas[0]:
assigned threads: 2
uptime: 47944824913
dss allocation precedence: "secondary"
decaying: time npages sweeps madvises purged
dirty: 1000 0 0 0 0
muzzy: 1000 0 0 0 0
allocated nmalloc (#/sec) ndalloc (#/sec) nrequests (#/sec) nfill (#/sec) nflush (#/sec)
small: 1336 9 0 4 0 9 0 0 0 0 0
large: 7487488 228604 4863 228601 4863 228604 4863 228604 4863 0 0
total: 7488824 228613 4864 228605 4863 228613 4864 228604 4863 0 0
active: 7532544
mapped: 13824000
retained: 2953216
base: 2375504
internal: 65536
metadata_thp: 0
tcache_bytes: 0
tcache_stashed_bytes: 0
resident: 9912320
abandoned_vm: 0
extent_avail: 0
n_lock_ops (#/sec) n_waiting (#/sec) n_spin_acq (#/sec) n_owner_switch (#/sec) total_wait_ns (#/sec) max_wait_ns max_n_thds
large 2 0 0 0 0 0 1 0 0 0 0 0
extent_avail 9 0 0 0 0 0 3 0 0 0 0 0
extents_dirty 457210 9727 0 0 0 0 5 0 0 0 0 0
extents_muzzy 8 0 0 0 0 0 3 0 0 0 0 0
extents_retained 9 0 0 0 0 0 3 0 0 0 0 0
decay_dirty 2355 50 0 0 0 0 135 2 0 0 0 0
decay_muzzy 2354 50 0 0 0 0 135 2 0 0 0 0
base 17 0 0 0 0 0 3 0 0 0 0 0
tcache_list 3 0 0 0 0 0 1 0 0 0 0 0
hpa_shard 0 0 0 0 0 0 0 0 0 0 0 0
hpa_shard_grow 0 0 0 0 0 0 0 0 0 0 0 0
hpa_sec 0 0 0 0 0 0 0 0 0 0 0 0
bins: size ind allocated nmalloc (#/sec) ndalloc (#/sec) nrequests (#/sec) nshards curregs curslabs nonfull_slabs regs pgs util nfills (#/sec) nflushes (#/sec) nslabs nreslabs (#/sec) n_lock_ops (#/sec) n_waiting (#/sec) n_spin_acq (#/sec) n_owner_switch (#/sec) total_wait_ns (#/sec) max_wait_ns max_n_thds
8 0 8 2 0 1 0 2 0 1 1 1 0 512 1 0.001 0 0 0 0 1 0 0 6 0 0 0 0 0 1 0 0 0 0 0
16 1 16 1 0 0 0 1 0 1 1 1 0 256 1 0.003 0 0 0 0 1 0 0 4 0 0 0 0 0 1 0 0 0 0 0
32 2 32 4 0 3 0 4 0 1 1 1 0 128 1 0.007 0 0 0 0 1 0 0 8 0 0 0 0 0 1 0 0 0 0 0
---
640 17 1280 2 0 0 0 2 0 1 2 1 0 32 5 0.062 0 0 0 0 1 0 0 5 0 0 0 0 0 1 0 0 0 0 0
---
large: size ind allocated nmalloc (#/sec) ndalloc (#/sec) nrequests (#/sec) curlextents
---
65536 44 65536 1 0 0 0 1 0 1
81920 45 81920 1 0 0 0 1 0 1
---
7340032 71 7340032 228602 4863 228601 4863 228602 4863 1
---
extents: size ind ndirty dirty nmuzzy muzzy nretained retained ntotal total
---
1048576 27 0 0 0 0 1 1044480 1 1044480
---
2097152 31 0 0 0 0 1 1908736 1 1908736
---
Bytes in small extent cache: 0
HPA shard stats:
Purge passes: 0 (0 / sec)
Purges: 0 (0 / sec)
Hugeifies: 0 (0 / sec)
Dehugifies: 0 (0 / sec)
In full slabs:
npageslabs: 0 huge, 0 nonhuge
nactive: 0 huge, 0 nonhuge
ndirty: 0 huge, 0 nonhuge
nretained: 0 huge, 0 nonhuge
In empty slabs:
npageslabs: 0 huge, 0 nonhuge
nactive: 0 huge, 0 nonhuge
ndirty: 0 huge, 0 nonhuge
nretained: 0 huge, 0 nonhuge
size ind npageslabs_huge nactive_huge ndirty_huge npageslabs_nonhuge nactive_nonhuge ndirty_nonhuge nretained_nonhuge
---
--- End jemalloc statistics ---print_stats=true, for 7MB+, run about 20s
___ Begin jemalloc statistics ___
Version: "5.3.0-0-g54eaed1d8b56b1aa528be3bdd1877e59c56fa90c"
Build-time option settings
config.cache_oblivious: true
config.debug: false
config.fill: true
config.lazy_lock: true
config.malloc_conf: ""
config.opt_safety_checks: true
config.prof: true
config.prof_libgcc: true
config.prof_libunwind: false
config.stats: true
config.utrace: false
config.xmalloc: true
Run-time option settings
opt.abort: false
opt.abort_conf: false
opt.cache_oblivious: true
opt.confirm_conf: false
opt.retain: true
opt.dss: "secondary"
opt.narenas: 1
opt.percpu_arena: "disabled"
opt.oversize_threshold: 16777216
opt.hpa: false
opt.hpa_slab_max_alloc: 65536
opt.hpa_hugification_threshold: 1992294
opt.hpa_hugify_delay_ms: 10000
opt.hpa_min_purge_interval_ms: 5000
opt.hpa_dirty_mult: "0.25"
opt.hpa_sec_nshards: 4
opt.hpa_sec_max_alloc: 32768
opt.hpa_sec_max_bytes: 262144
opt.hpa_sec_bytes_after_flush: 131072
opt.hpa_sec_batch_fill_extra: 0
opt.metadata_thp: "disabled"
opt.mutex_max_spin: 600
opt.background_thread: true (background_thread: true)
opt.dirty_decay_ms: 1000 (arenas.dirty_decay_ms: 1000)
opt.muzzy_decay_ms: 1000 (arenas.muzzy_decay_ms: 1000)
opt.lg_extent_max_active_fit: 6
opt.junk: "false"
opt.zero: false
opt.xmalloc: false
opt.experimental_infallible_new: false
opt.tcache: false
opt.tcache_max: 32768
opt.tcache_nslots_small_min: 20
opt.tcache_nslots_small_max: 200
opt.tcache_nslots_large: 20
opt.lg_tcache_nslots_mul: 1
opt.tcache_gc_incr_bytes: 65536
opt.tcache_gc_delay_bytes: 0
opt.lg_tcache_flush_small_div: 1
opt.lg_tcache_flush_large_div: 1
opt.thp: "default"
opt.prof: false
opt.prof_prefix: "jeprof"
opt.prof_active: true (prof.active: false)
opt.prof_thread_active_init: true (prof.thread_active_init: false)
opt.lg_prof_sample: 19 (prof.lg_sample: 0)
opt.prof_accum: false
opt.lg_prof_interval: -1
opt.prof_gdump: false
opt.prof_final: false
opt.prof_leak: false
opt.prof_leak_error: false
opt.stats_print: true
opt.stats_print_opts: ""
opt.stats_print: true
opt.stats_print_opts: ""
opt.stats_interval: -1
opt.stats_interval_opts: ""
opt.zero_realloc: "free"
Profiling settings
prof.thread_active_init: false
prof.active: false
prof.gdump: false
prof.interval: 0
prof.lg_sample: 0
Arenas: 2
Quantum size: 16
Page size: 4096
Maximum thread-cached size class: 32768
Number of bin size classes: 36
Number of thread-cache bin size classes: 41
Number of large size classes: 196
Allocated: 8537400, active: 8581120, metadata: 2441040 (n_thp 0), resident: 10960896, mapped: 14872576, retained: 4001792
Count of realloc(non-null-ptr, 0) calls: 0
Background threads: 1, num_runs: 20454, run_interval: 1662155 ns
n_lock_ops (#/sec) n_waiting (#/sec) n_spin_acq (#/sec) n_owner_switch (#/sec) total_wait_ns (#/sec) max_wait_ns max_n_thds
background_thread 5 0 0 0 0 0 1 0 0 0 0 0
max_per_bg_thd 20537 604 0 0 0 0 4 0 0 0 0 0
ctl 2 0 0 0 0 0 1 0 0 0 0 0
prof 0 0 0 0 0 0 0 0 0 0 0 0
prof_thds_data 0 0 0 0 0 0 0 0 0 0 0 0
prof_dump 0 0 0 0 0 0 0 0 0 0 0 0
prof_recent_alloc 0 0 0 0 0 0 0 0 0 0 0 0
prof_recent_dump 0 0 0 0 0 0 0 0 0 0 0 0
prof_stats 0 0 0 0 0 0 0 0 0 0 0 0
arenas[0]:
assigned threads: 2
uptime: 34005585120
dss allocation precedence: "secondary"
decaying: time npages sweeps madvises purged
dirty: 1000 0 0 20453 41908197
muzzy: 1000 0 0 0 0
allocated nmalloc (#/sec) ndalloc (#/sec) nrequests (#/sec) nfill (#/sec) nflush (#/sec)
small: 1336 9 0 4 0 9 0 0 0 0 0
large: 8536064 20456 601 20453 601 20456 601 20456 601 0 0
total: 8537400 20465 601 20457 601 20465 601 20456 601 0 0
active: 8581120
mapped: 14872576
retained: 4001792
base: 2375504
internal: 65536
metadata_thp: 0
tcache_bytes: 0
tcache_stashed_bytes: 0
resident: 10960896
abandoned_vm: 0
extent_avail: 0
n_lock_ops (#/sec) n_waiting (#/sec) n_spin_acq (#/sec) n_owner_switch (#/sec) total_wait_ns (#/sec) max_wait_ns max_n_thds
large 2 0 0 0 0 0 1 0 0 0 0 0
extent_avail 40915 1203 0 0 0 0 3 0 0 0 0 0
extents_dirty 40914 1203 0 0 0 0 3 0 0 0Source: jemalloc/jemalloc