Baike.dev
Log in
Back to tool/Back to issues
#79130·starrocks

[Bug] BE crashes with SIGSEGV in CompactionManager::submit_compaction_task during batch full-load workload on v3.2.16

Author: littlebutteCreated Sep 15, 2026Updated Sep 15, 2026
Labelstype/bug
Image Image Image Image

be.out.txt

StarRocks Version

StarRocks 3.2.16 RELEASE

Build: 8dea52d

Deployment

  • Deployment mode: Shared-nothing
  • FE nodes: 3
  • BE nodes: 3
  • BE hardware:
    • CPU: 24 cores
    • Memory: 64 GB
    • Swap: disabled
  • Storage path:
    • /data/storage
  • BE storage page cache:
    • enabled
    • storage_page_cache_limit = 4294967296 (4 GB)

Workload

We use DataX to perform scheduled full data synchronization into StarRocks.

The workload characteristics are:

  • About 500 tables are synchronized.
  • The synchronization starts around midnight.
  • A large number of tables are loaded during the same batch window.
  • The issue has occurred on different BE nodes on different days.
  • The affected BE is not fixed:
    • one occurrence happened on BE6
    • another occurrence happened on BE5

The BE crashes only appeared during this batch full-load workload.

Problem Description

During the nightly DataX full synchronization, one BE process unexpectedly crashes.

The failed BE is not always the same node.

We initially suspected OOM or resource exhaustion, but after checking the OS logs and Grafana metrics, we did not find evidence of Linux OOM, CPU saturation, memory exhaustion, or disk I/O saturation.

The latest crash was captured in be.out.

The BE terminated with SIGSEGV while executing:

CompactionManager::_schedule() -> CompactionManager::submit_compaction_task() -> google::LogMessage::Init() -> operator new() -> malloc -> je_malloc_default -> SIGSEGV

Crash Stack

3.2.16 RELEASE (build 8dea52d)

query_id:00000000-0000-0000-0000-000000000000
fragment_instance:00000000-0000-0000-0000-000000000000

tracker:process consumption: 14542993792
tracker:jemalloc_metadata consumption: 2054090736
tracker:query_pool consumption: 746208384
tracker:query_pool/connector_scan consumption: 0
tracker:load consumption: 40656
tracker:metadata consumption: 977398409
tracker:tablet_metadata consumption: 27643210
tracker:rowset_metadata consumption: 125055768
tracker:segment_metadata consumption: 25114856
tracker:column_metadata consumption: 799584575
tracker:tablet_schema consumption: 1485210
tracker:segment_zonemap consumption: 12562015
tracker:short_key_index consumption: 2833673
tracker:column_zonemap_index consumption: 31341199
tracker:ordinal_index consumption: 364733776
tracker:bitmap_index consumption: 0
tracker:bloom_filter_index consumption: 0
tracker:compaction consumption: 552434568
tracker:schema_change consumption: 0
tracker:column_pool consumption: 0
tracker:page_cache consumption: 4316669456
tracker:update consumption: 853128215
tracker:chunk_allocator consumption: 2147725712
tracker:passthrough consumption: 0
tracker:clone consumption: 0
tracker:consistency consumption: 12378176
tracker:datacache consumption: 0
tracker:replication consumption: 0

*** Aborted at 1789412112 (unix time) try "date -d @1789412112" if you are using GNU date ***
PC: @          0x7851e50 (unknown)

*** SIGSEGV (@0x50) received by PID 77546
(TID 0x7ff10dff2700) from PID 80; stack trace: ***

    @          0x6d79c02 google::(anonymous namespace)::FailureSignalHandler()
    @     0x7ff252cc0bba os::Linux::chained_handler()
    @     0x7ff252cc61ce JVM_handle_linux_signal
    @     0x7ff252cba0c8 signalHandler()
    @     0x7ff251e97630 (unknown)
    @          0x7851e50 (unknown)
    @          0x77e1ada je_malloc_default
    @          0x2f92348 malloc
    @          0x9141765 operator new()
    @          0x6d6c801 google::LogMessage::Init()
    @          0x564c4e4 starrocks::CompactionManager::submit_compaction_task()
    @          0x564ee6c starrocks::CompactionManager::_schedule()
    @          0x91ba6c0 execute_native_thread_routine
    @     0x7ff251e8fea5 start_thread
    @     0x7ff251290b0d __clone
    @                0x0 (unknown)

Memory Status Before Crash

According to the BE memory tracker immediately before the crash:

process          = 14,542,993,792 bytes (~13.5 GiB)
query_pool       =    746,208,384 bytes
load             =         40,656 bytes
compaction       =    552,434,568 bytes
page_cache       =  4,316,669,456 bytes
update           =    853,128,215 bytes
chunk_allocator  =  2,147,725,712 bytes

Each BE has 64 GB physical memory.

Therefore, the BE process was using approximately 13.5 GiB when the SIGSEGV occurred.

We did not observe the BE approaching the physical memory limit.

OS OOM Check

We checked the operating system logs for OOM events.

No Linux OOM killer event was found around the crash time.

We did not find messages similar to:

Out of memory
Killed process starrocks_be
oom-kill

Therefore, currently we do not believe the BE process was terminated by the Linux OOM killer.

Grafana Observations

We narrowed the Grafana monitoring window to approximately:

2026-09-15 02:45 - 03:05

Immediately before the affected BE disappeared from monitoring:

CPU

CPU was not saturated.

At approximately 02:55:

BE4 CPU Idle: ~78.9%
BE5 CPU Idle: ~79.1%
BE6 CPU Idle: ~79.7%

The affected BE therefore had only about 21% CPU utilization.

BE Memory

The affected BE process memory was approximately 13-14 GiB before the crash.

No large memory spike close to 64 GB was observed.

The BE memory metric stopped being reported shortly after the crash.

Loading Memory

Loading memory was very low immediately before the crash.

We did not observe a large loading-memory spike.

Disk I/O

Disk I/O utilization on the affected BE was generally below 50% immediately before the crash.

Other healthy BE nodes reached higher I/O utilization without crashing.

Therefore, disk utilization did not appear saturated at the time of the crash.

Compaction Observations

Compaction activity was significant during the nightly DataX load.

BE Max Compaction Score was generally around 20-60, with occasional values above 70 on some BE nodes.

Around 02:49:

BE4: 52
BE5: 61   <-- BE that later crashed
BE6: 42

Another healthy BE reached a higher Max Compaction Score (~74) without crashing.

Therefore, we do not currently believe the crash was simply caused by an excessively high compaction score.

However, the crash stack clearly shows that the crashing thread was executing:

CompactionManager::_schedule()
    -> CompactionManager::submit_compaction_task()

This makes the compaction scheduling path particularly suspicious.

Reproduction Pattern

We do not yet have a deterministic minimal reproduction.

The issue has occurred during the nightly DataX batch full synchronization.

Typical workload:

~500 tables
    ->
batch/full synchronization
    ->
many concurrent writes
    ->
rowset/segment generation
    ->
continuous compaction activity
    ->
one BE occasionally crashes

The affected BE is not fixed.

For example:

Previous occurrence: BE6 crashed
Latest occurrence:   BE5 crashed

This makes a single-node hardware issue less likely.

Expected Behavior

BE should continue running normally under batch loading and compaction workload.

Even under heavy compaction pressure, the BE should not terminate with SIGSEGV.

Actual Behavior

One BE randomly terminates with SIGSEGV during the batch full-load window.

The crash occurs in the CompactionManager scheduling path.

After the BE crashes, FE marks the BE as DEAD and the DataX synchronization is affected.

Restarting the BE restores the node.

Initial Analysis

Based on the current evidence:

  1. Linux OOM was not observed.
  2. BE process memory was only about 13.5 GiB on a 64 GB machine.
  3. CPU was not saturated.
  4. Disk I/O utilization was not saturated.
  5. Loading memory was low.
  6. Compaction activity was present.
  7. The crash was SIGSEGV rather than a memory-limit error.
  8. The crashing thread was inside CompactionManager::submit_compaction_task().

The stack contains:

je_malloc_default
malloc
operator new()
google::LogMessage::Init()
CompactionManager::submit_compaction_task()
CompactionManager::_schedule()

Because the SIGSEGV occurs inside the malloc/logging path, we are also wondering whether this could be caused by memory corruption, use-after-free, or another concurrency/lifetime issue that occurred earlier.

However, we do not currently have enough evidence to confirm the root cause.

Possibly Related Issue

We found the following issue:

#73415 "BE crashes with SIGSEGV in multiple locations due to use-after-free in TabletUpdates and ColumnReader — 3.5.16 shared-nothing cluster"

There are some similarities:

  • shared-nothing deployment
  • 3 BE nodes
  • SIGSEGV
  • compaction-related background threads
  • crashes can occur on different BE nodes
  • workload involving continuous/heavy writes

However, our environment is StarRocks 3.2.16 and our stack is different:

CompactionManager::submit_compaction_task()
CompactionManager::_schedule()

Therefore, we are not sure whether these issues are related.

Questions

Could you please help confirm:

  1. Is this a known bug in StarRocks 3.2.16?

  2. Are there known race conditions, use-after-free issues, or heap corruption bugs around:

CompactionManager::submit_compaction_task()
CompactionManager::_schedule()

in the 3.2 branch?

  1. Has this specific crash path been fixed in a later StarRocks version?

  2. Which StarRocks version would you recommend upgrading to for this issue?

  3. Are there any additional logs, core dumps, gdb backtraces, or metrics that would help identify the root cause?

We can provide additional BE logs and Grafana screenshots if needed.

Additional Information

The affected BE configuration is mostly default.

Relevant configuration:

storage_root_path = /data/storage

disable_storage_page_cache = false
storage_page_cache_limit = 4294967296

We are currently considering reducing the DataX batch concurrency as a temporary mitigation, but would like to determine whether the SIGSEGV corresponds to a known StarRocks bug.

Image Image Image Image

Source: StarRocks/starrocks

View original on GitHubView discussion on GitHub