#159349 ES|QL: FallbackSyntheticSourceBlockLoader 1KB breaker estimate misses _ignored_source binary doc-values (OOME)

Elasticsearch Version

9.6.0 (serverless search tier)

Installed Plugins

default serverless distribution

Java Version

bundled OpenJDK 26.0.2+10-55

OS Version

Amazon Linux 2023 aarch64 (6.12.92-122.166.amzn2023.aarch64)

Problem Description

ES|QL can still OutOfMemoryError: Java heap space a 4Gi search heap while loading many unmapped fields through fallback synthetic _source, even though field-reader circuit-breaker accounting from #140666 is present.

On a production serverless observability search node (2 search replicas × 4Gi heap), both replicas OOM'd twice ~8 minutes apart. Eclipse MAT on the heap dump:

13,838 instances of FallbackSyntheticSourceBlockLoader$IgnoredSourceRowStrideReader retain 3,003,319,072 bytes (71.99%) of a 3.9GB used / 4GB heap
those readers cover 4,629 distinct unmapped field names (high-cardinality app.*-style fields stored in _ignored_source)
also live: 10,051 FieldAttribute, 14,310 ValuesSourceReaderOperator$FieldWork, 13,841 SourceFilter
hottest threads are esql_worker, all in:

AbstractTSDBDocValuesProducer$BinaryDecoder.decode
  → MultiValuedSortedBinaryDocValues.nextValue
  → IgnoredSourceFieldMapper.loadIgnoredFields
  → FallbackSyntheticSourceBlockLoader$IgnoredSourceRowStrideReader.read
  → ValuesFromSingleReader / ValuesSourceReaderOperator
  → Driver.run  (esql_worker)

The JVM [OOM] allocation stack was a stateless_prewarm blob-cache fill (~122KB retained). That is the thread that lost the last allocation, not the accumulator.

HierarchyCircuitBreakerService did try to shed memory (attempting to trigger G1GC due to high heap usage [4180136504], then GC did not bring memory usage down). G1 cannot reclaim the ES|QL working set because the _ignored_source byte[] are still referenced by the live readers.

Why #140218 / #140666 are not sufficient here

#140666 charges a flat BlockSourceReader.ESTIMATED_SIZE (1KB) per IgnoredSourceRowStrideReader:

breaker.addEstimateBytesAndMaybeBreak(ESTIMATED_SIZE, "load blocks");

13,838 readers × 1KB ≈ 13.5MB accounted vs 3.00GB retained (~220× undercount, ~217KB/reader). The missing bytes are decompressed _ignored_source binary doc-values (byte[] dominated by MultiValuedSortedBinaryDocValues$SeparateCounts / TSDB BinaryDecoder), not ValuesSourceReaderOperator.FieldWork / BlockSourceReader.scratch from #140218.

#140666 merged 2026-02-21; this dump is 9.6.0 (2026-09-04). #140218 was closed 2026-03-11 as “fixes merged, keep monitoring heap-attack tests.” This is a remaining gap on the synthetic _ignored_source binary DV path added around #143784.

Related but not this bug: #122428 (perf of SourceValueFetcher parsing a full source Map).

Steps to Reproduce

Not reduced to a yaml/IT yet. Shape that matches the dump:

Logsdb / synthetic-source index (or data stream) with thousands of unmapped fields persisted in _ignored_source as binary doc-values.
ES|QL query that loads those fields (wide KEEP / column explosion / unmapped_fields=LOAD), enough to open ~O(fields × leaves) IgnoredSourceRowStrideReaders on a 4Gi heap.
Expectation: CircuitBreakingException before JVM OOME.
Actual: G1 humongous-allocation death spiral, then java.lang.OutOfMemoryError: Java heap space.

A heap-attack variant of #140218 that uses synthetic source + _ignored_source binary DV (rather than mapped keyword/text) should fail today.

Logs (if relevant)

attempting to trigger G1GC due to high heap usage [4180136504]
attempt to trigger young GC failed to bring memory down, triggering full GC
GC did not bring memory usage down, before [4180136504], after [4196219992]
java.lang.OutOfMemoryError: Java heap space

GC: G1 Humongous Allocation / Evacuation Failure: Allocation on 4096M heap, Full GC ~4021M→4014M.

Suggested fix

Account for the live _ignored_source binary doc-values (and/or decompressed BytesRef/byte[]) on IgnoredSourceRowStrideReader, not only the 1KB constructor estimate. Heap-attack coverage for “many unmapped fields on synthetic source”.

Relates #140218 #140666 #143784