Heap corruption / SIGABRT during SST rejoin after upgrade from 29.0.1 to 29.2.2

Bug Description:

glibc allocator aborts inside Manticore HTTP parser (HttpRequestParser_c::FinishParserKeyVal / Reinit) while concurrent SST traffic runs.

searchd.log.gz searchd.log.1.gz

After upgrading one node in a three-node replication cluster from Manticore 29.0.1 to 29.2.2, JOIN CLUSTER required a full SST because the joiner's local state UUID was zero. During SST, the upgraded searchd repeatedly aborted with:

malloc(): unaligned tcache chunk detected
Handling signal 6

systemd restarted the process four times. The donor repeatedly restarted/canceled state transfer. The final process stayed active, but the join did not complete: the upgraded node remained disconnected / joining, with cluster_size=0 and retries limit exceeded.

This looks like heap corruption detected in the HTTP request parser while SST progress/API traffic is concurrent. Different ordinary client requests happened to be executing when each abort was detected, so those queries may be victims rather than the original corruptor.

Environment

Joiner:

OS: Ubuntu 24.04.4 LTS, x86_64
Kernel: Linux 6.8.0-136-generic
Manticore: 29.2.2 fe8f84c8b@26082106 dev
Package: manticore 29.2.2-26082106-fe8f84c8b
Columnar: 13.10.0 5f1b59b@26081915
manticore-server-core-dbgsym: 29.2.2-26082106-fe8f84c8b
manticore-columnar-lib-dbgsym: 13.10.0+26081918-5f1b59b5
manticore-galera-dbgsym: 3.37

Both synced donor nodes were still on:

Manticore: 29.0.1 2f9e131f1@26081222 dev
Package: manticore 29.0.1-26081222-2f9e131f1
Columnar: 13.9.0 dbc40aa@26080509
manticore-server-core-dbgsym: 29.0.1-26081222-2f9e131f1
manticore-columnar-lib-dbgsym: 13.9.0+26080512-dbc40aa7
manticore-galera-dbgsym: 3.37

Upgrade recorded by APT:

2026-08-23 13:27:01-13:27:05 EDT
manticore: 29.0.1-26081222-2f9e131f1 -> 29.2.2-26082106-fe8f84c8b
manticore-server-core-dbgsym: 29.0.1-26081222-2f9e131f1 -> 29.2.2-26082106-fe8f84c8b
manticore-tools-dbgsym: 29.0.1-26081222-2f9e131f1 -> 29.2.2-26082106-fe8f84c8b
manticore-columnar-lib-dbgsym: 13.9.0+26080512-dbc40aa7 -> 13.10.0+26081918-5f1b59b5

Joiner service command:

/usr/bin/searchd --config /etc/manticoresearch/manticore.conf --nodetach --coredump --logreplication --force-preread

Relevant settings/state:

threads = 8
SQL port = 9306
API/cluster join port = 9312
replication listen range = 9315-9325
runtime log_level = replication
gcache.page_size = 128M
gcache.size = 144M
replicated tables = 9

Resource pressure was not present at inspection time:

RAM: 125 GiB total, 81 GiB available
Swap: 1 GiB total, ~2 MiB used
Disk: 243 GiB available
Inodes: 3% used
Kernel log: no OOM kill, kernel segfault, or killed-process event in crash window

Steps to reproduce

1. Start with a healthy three-node cluster on Manticore 29.0.1.

2. Remove/stop one node for an in-place rolling upgrade.

3. Upgrade that node to Manticore 29.2.2 while the two donors remain on 29.0.1.

4. Start the upgraded node with --coredump --logreplication --force-preread.

5. Rejoin it using a synced 29.0.1 donor:

   sql复制

   JOIN CLUSTER jobsite AT '<donor-host>:9312';

6. Because local state is empty (00000000-0000-0000-0000-000000000000:-1), Manticore switches from IST to SST for nine RT tables.

7. Keep normal production SQL/SphinxAPI/HTTP traffic running while SST transfers/activates tables.

Observed reproduction on this run:

13:38:47 first join begins; SST selected
13:39:39 SST canceled and cluster deleted by retry workflow
13:39:41 second join begins; SST selected
13:39:53 SST canceled and cluster deleted by retry workflow
13:40:55 third join begins; SST selected
13:43:07 first SIGABRT
13:43:53 second SIGABRT after systemd restart and automatic SST retry
13:45:44 third SIGABRT
13:46:07 fourth SIGABRT
13:46:09 final automatic restart

Expected behavior

SST completes, the upgraded node becomes synced, and ordinary client requests continue safely during state transfer.

Actual behavior

Each affected process reports heap corruption and aborts with signal 6. Representative journal/log output:

malloc(): unaligned tcache chunk detected
------- FATAL: CRASH DUMP -------
Handling signal 6
systemd: Main process exited, code=dumped, status=6/ABRT
systemd: Failed with result 'core-dump'.

Representative backtraces:

malloc
operator new(unsigned long)
CSphOrderedHash<...>::AddImpl(...)
HttpRequestParser_c::FinishParserKeyVal()
HttpRequestParser_c::cbParserHeaderField(...)
http_parser_execute
HttpRequestParser_c::ParseHeader(...)
HttpServe(...)

__libc_free
HttpRequestParser_c::Reinit()
HttpServe(...)
MultiServe(...)

The crash dump's request attribution varied between invalid query, a SphinxAPI request, and SHOW TABLES, which suggests allocator corruption is detected by whichever request thread allocates/frees next.

Donor-side consequence:

State transfer to -1.-1 (left the group) failed: -125 (Operation canceled)
State transfer to 1.0 (...) failed: -125 (Operation canceled)
sending install message failed: Resource temporarily unavailable

Final joiner status after last restart:

cluster_jobsite_status = disconnected
cluster_jobsite_node_state = joining
cluster_jobsite_state_uuid = 00000000-0000-0000-0000-000000000000
cluster_jobsite_state_seqno = -1
cluster_jobsite_size = 0
cluster_jobsite_last_error = retries limit exceeded
cluster_jobsite_sst_stage = analyze remote

Both 29.0.1 donor nodes remained active, primary, and synced without restarting.

After the aborted SST attempts, the joiner also logged Galera cache allocation failures with gcache.page_size=128M / gcache.size=144M. Host RAM and disk remained available. These may be a secondary symptom:

Failed to allocate ~21-23 MB; space left ~17 MB; total allocated ~117 MB

Questions

1. Is SST/rejoin from 29.0.1 donors to a 29.2.2 joiner supported during a rolling upgrade?
2. Is this a known 29.2.2 regression in HttpRequestParser_c or concurrent SST progress handling?
3. Is there a safe workaround (for example disabling SST progress logging or preventing HTTP/API traffic during SST) while retaining replication logging?
4. Should gcache.size / gcache.page_size be changed for this data volume, or are those allocation errors only consequences of the failed transfers?

Attachments

• searchd.log — current receiver log; contains later crashes and final failed join state.
• searchd.log.1 — rotated receiver log; contains earlier crashes and full backtraces.
• searchd-29.2.2-fe8f84c8b — exact crashing binary.
• searchd-29.2.2-fe8f84c8b.debug — matching debug symbols from manticore-server-core-dbgsym.

An Ubuntu Apport crash file with embedded core is also available (Signal: 6, about 314 MB) if needed.

GitHub limits ordinary issue attachments to 25 MB. Uploaded these prepared files:

searchd.log.gz                                      7.0 MB
searchd.log.1.gz                                     89 KB
searchd-29.2.2-fe8f84c8b.gz                          11 MB
searchd-29.2.2-fe8f84c8b.debug.part-aa.gz            22 MB
searchd-29.2.2-fe8f84c8b.debug.part-ab.gz            16 MB
searchd-29.2.2-fe8f84c8b.debug.part-ac.gz           2.6 MB

All prepared files were uploaded; debug symbols are split into three gzip parts.

Reassemble matching symbols after downloading all three parts:

gzip -dc searchd-29.2.2-fe8f84c8b.debug.part-*.gz > searchd-29.2.2-fe8f84c8b.debug

Manticore Search Version:

29.2.2

Operating System Version:

Ubuntu 24.04

Have you tried the latest development version?

Yes — crash occurred on 29.2.2 fe8f84c8b@26082106 dev.

Internal Checklist:

searchd-29.2.2-fe8f84c8b.debug.part-ac.gz searchd.log.gz searchd.log.1.gz