#1007·memray

Make hook-reachable C++ independent from the process allocator

Author: pablogsalCreated Aug 28, 2026Updated Aug 28, 2026
Labelsbug

The allocator hooks can allocate through the allocator they are currently intercepting. This normally works, but it can deadlock if the hook was entered while malloc/jemalloc/tcmalloc/etc. was holding one of its internal locks.

libhugetlbfs is a good way to expose it, but this is not really a huge-pages-specific bug. On older glibc, malloc can call MORECORE while the arena is locked (glibc source). libhugetlbfs replaces that with an implementation which calls mmap (source), so we can get:

malloc -> allocator lock -> mmap -> Memray mmap hook
       -> C++ allocation -> malloc -> same lock forever

We have already seen the same shape with jemalloc in #669: jemalloc called mmap while creating an arena and Memray created a std::vector from the mmap hook, which went back into jemalloc. Fixing that one vector does not prevent another string, map, TLS object, exception, or library call from reintroducing the problem.

A recursion guard does not solve this. It can stop us recording the nested allocation, but the real nested malloc still tries to take the lock which is already held.

The invariant should be that Memray-owned dynamic C++ storage reachable from an allocation/deallocation hook never uses the process allocator. This must not change the current profiling behavior: in particular, native stacks for mmap events stay enabled.

There are several small implementations we can borrow from: allocator-aware STL aliases backed by a private pool, like the LLVM sanitizer allocator, gperftools LowLevelAlloc, or Perfetto's unhooked allocator. We should not globally replace new/delete, because allocations crossing shared-library boundaries can then be freed by the wrong heap.

We also need a hostile regression test which holds a fake allocator lock across mmap/munmap and fails if Memray-owned hook code calls malloc, calloc, realloc, or free. That gives us one generic check instead of fixing this allocator by allocator or container by container.

Libunwind can introduce the same inversion through the loader lock (gperftools example). Its unw_set_iterate_phdr_function API lets us give it an immutable program-header snapshot, so native mmap stacks can stay enabled without calling dl_iterate_phdr from the allocation hook.