[Bug] [Bug] first_value/nth_value return corrupted values and can crash the BE once a window partition spans more than 257 blocks

Search before asking

• I had searched in the issues and found no similar issues.

Version

Affected: Apache Doris 4.1.4 — BE reports doris-4.1.4-rc04-ad35a140c7f (the newest released build; images apache/doris:fe-4.1.4 + be-4.1.4, pushed 2026-09-11). Single FE + single BE, Nereids planner, otherwise default configuration, Linux x86_64.

Also affected: current master. There is no nightly/master Doris image, so I built apache/doris master commit c4dee4bd5e82c6f0f380e0243cbaf265999e6a6a from source with apache/doris:build-env-ldb-toolchain-latest and ran the identical battery against a single-FE/single-BE cluster from that build. The clone's HEAD was verified equal to GitHub's master HEAD (2026-09-17T03:34:48Z). BE reports doris-0.0.0-Unknown because a local build carries no version stamp; the shipped default batch_size is 8160 there too.

master reproduces every measurement identically to 4.1.4 — not approximately, the same counts row for row:

and master crashed the BE during the run. Because it is a locally built binary the trace carries full source lines, which is the clearest statement of the bug available:

*** SIGSEGV invalid permissions for mapped object (@0x7f15c87fd000) received by PID 2333 ***
 2# memcpy at ./be/build_Release/../src/glibc-compatibility/memcpy/memcpy_x86_64.cpp:219
 3# doris::ColumnStr<unsigned int>::insert_from(doris::IColumn const&, unsigned long)
      at ../src/core/column/column_string.h:188
 4# doris::IAggregateFunction::insert_result_into_range(char const*, doris::IColumn&, unsigned long, unsigned long) const
      at ../src/exprs/aggregate/aggregate_function.h
 5# doris::AnalyticSinkLocalState::_insert_result_info(long, long)
      at ./be/src/exec/operator/analytic_sink_operator.cpp:404
 6# doris::AnalyticSinkLocalState::_get_next_for_unbounded_rows(long, long)
      at ./be/src/exec/operator/analytic_sink_operator.cpp:248
 8# doris::AnalyticSinkOperatorX::sink_impl(doris::RuntimeState*, doris::Block*, bool)
      at ./be/src/exec/operator/analytic_sink_operator.cpp:759

A memcpy inside ColumnStr::insert_from, reached from the analytic sink writing out an aggregate's saved value — i.e. copying a string through an offset that no longer describes the row it was saved for.

No fix is in flight: the most recent commit touching analytic_sink_operator.cpp is #67274 (2026-08-31) and the most recent touching aggregate_function_window.h is #64864 (2026-06-26); neither addresses the saved-pointer invalidation.

What's Wrong?

first_value() over a window silently returns values that are not in the window — and on a VARCHAR column, values that are not in the table at all. On the same data the BE can also abort the query with a bogus 4 GB string length, or crash outright.

The trigger is that the analytic operator evicts input rows it believes are no longer needed, while first_value is still holding a raw pointer plus an absolute row offset into exactly the columns being evicted.

1. first_value(varchar) returns a string that is nowhere in the table

With PARTITION BY c, first_value(c) must equal the row's own c — every partition contains one distinct value, so this is an exact oracle needing no reference engine. 20 000 rows, every value at most 8 characters:

A sample row: c = 'xyz', so the correct answer is 'xyz', and Doris returns

azczazaczcbczcbczcbczyaxabxxaaxbbxabaxabcayabxacazayxaxcxyzxaxyxxxx…   (229 characters)

That is a run of adjacent values read out of the string column through a wrong offset/length pair — the correct 3-character value is visible inside it.

2. first_value(int) is wrong for most of the table

Same shape on an INT column, ground truth computed in Python (PARTITION BY e with e boolean, so two partitions of ~10 000 rows):

No warning, no error — the query simply returns wrong numbers for most rows.

3. It reproduces at the shipped batch_size on an ordinary table

batch_size only makes it cheap to demonstrate. What actually matters is the partition size in blocks, so the same defect is reached with batch_size left completely alone (shipped default 8160) by making one partition large — the threshold being 257 * 8160 = 2 097 120 rows. Single partition, nothing configured:

The boundary lands exactly where the 257 * batch_size rule predicts (1 280 000 < 2 097 120 < 2 560 000), and raising batch_size to 65535 moves the threshold out of reach again, which isolates the block count as the cause.

One caveat worth knowing when reproducing: the block count is produced by the scan, so it depends on how the table is physically laid out, not only on batch_size. On a table I had built by repeated INSERT INTO … SELECT doubling (many small rowsets) the same 2 560 000-row partition was wrong at batch_size = 65535 as well, where a table of the same size written by a single INSERT … SELECT was clean at 65535. So a fragmented table reaches 257 blocks with fewer rows, and raising batch_size does not reliably protect it.

4. The exact trigger

The onset is not fuzzy. Corruption begins exactly when

partition_size  >  257 * batch_size

which is block_num = 256 in _remove_unused_rows(), plus the + 1 in its guard. Verified by fixing the partition size and walking batch_size across the predicted boundary:

Three for three, on the row exactly.

5. The same corruption as an error, and as a crash

nth_value(varchar, 3) over ROWS BETWEEN CURRENT ROW AND UNBOUNDED FOLLOWING, 20 000 rows, batch_size = 4:

errCode = 2, detailMessage = [E-3113] string column length is too large:
total_length=4294967303, element_number=4, rows=0

4294967303 is 2³² + 7 — the string length arithmetic wrapped a 32-bit counter, which is the same wrong offset/length pair that produces the 229-character first_value answers.

The same shape took the BE down on both builds — on 4.1.4 at batch_size = 1, and on the master build during the battery. Stack from the 4.1.4 BE log (the master one, with full source lines, is in the Version section):

2#  doris::ColumnStr<unsigned int>::insert_from(doris::IColumn const&, unsigned long)
       at be/src/core/column/column_string.h:71
3#  doris::IAggregateFunction::insert_result_into_range(char const*, doris::IColumn&, unsigned long, unsigned long) const
       at be/src/exprs/aggregate/aggregate_function.h:252
4#  doris::AnalyticSinkLocalState::_insert_result_info(long, long)
       at be/src/exec/operator/analytic_sink_operator.cpp:406
5#  doris::AnalyticSinkLocalState::_get_next_for_unbounded_rows(long, long)
       at be/src/exec/operator/analytic_sink_operator.cpp:250
6#  doris::AnalyticSinkLocalState::_execute_impl(doris::RuntimeState*)
7#  doris::AnalyticSinkOperatorX::sink_impl(doris::RuntimeState*, doris::Block*, bool)

Frame 2 is the aggregate's saved value being copied out: to.insert_from(*_ptr, _offset). It is reading through a pointer/offset that no longer describes the row it was saved for.

What You Expected?

first_value(c) OVER (PARTITION BY c …) must return the row's own c for every row, and first_value(a) must return the first a of the partition in window order — regardless of batch_size, partition size, or whether the operator decided to release memory mid-partition. The answer must not depend on how the input was split into vectorised blocks.

At minimum, a query must not silently return values that do not exist in the input, and must not crash the backend.

How to Reproduce?

Single FE + single BE, default configuration:

docker network create --subnet=172.30.0.0/24 doris-net
docker run -d --name fe --network doris-net --ip 172.30.0.2 \
  -e FE_SERVERS="fe1:172.30.0.2:9010" -e FE_ID=1 -p 9030:9030 apache/doris:fe-4.1.4
docker run -d --name be --network doris-net --ip 172.30.0.3 \
  -e FE_SERVERS="fe1:172.30.0.2:9010" -e BE_ADDR="172.30.0.3:9050" apache/doris:be-4.1.4

A. Smallest form — exact oracle, no reference engine needed. Load 20 000 rows of (id INT, c VARCHAR(255)) where c is a random string of at most 8 characters, then:

SET batch_size = 4;                     -- 20 000 rows / 4 = 5 000 blocks per scan
SELECT id, c,
       first_value(c) OVER (PARTITION BY c ORDER BY c, id
                            ROWS BETWEEN UNBOUNDED PRECEDING AND UNBOUNDED FOLLOWING) AS w
FROM t
HAVING w <> c;                          -- must be empty; returns rows whose w is 229 chars long

Because the window partitions by c, w can only be c. Any returned row is a wrong answer. SET batch_size = 65535 on the same table returns nothing, confirming the data is fine.

B. At the shipped batch_size, no session variable touched. Build (id INT, a INT, g BIGINT, e INT) with e = 0 on every row, so the whole table is one partition, and grow it past 257 * 8160 = 2 097 120 rows:

-- batch_size left at its shipped default of 8160
SELECT count(*) FROM (
  SELECT id, first_value(a) OVER (PARTITION BY e ORDER BY g, id
                                  ROWS BETWEEN UNBOUNDED PRECEDING AND UNBOUNDED FOLLOWING) AS w
  FROM big
) s
WHERE w <> (SELECT a FROM big ORDER BY g, id LIMIT 1);
-- 1 280 000 rows -> 0        (below 257*8160)
-- 2 560 000 rows -> 454 720  (must be 0)

SET batch_size = 65535 returns 0 for both, which isolates the block count as the cause.

C. The crash / 4 GB length. On the 20 000-row varchar table:

SET batch_size = 4;
SELECT id, nth_value(c, 3) OVER (PARTITION BY b % 13 ORDER BY c, id
                                 ROWS BETWEEN CURRENT ROW AND UNBOUNDED FOLLOWING) AS w
FROM t;
-- [E-3113] string column length is too large: total_length=4294967303

SET batch_size = 1;                      -- took the BE down in my run

A reproduction script with the ground-truth oracles for all of the above is attached to this report.

Anything Else?

No response

Are you willing to submit PR?

• Yes I am willing to submit a PR!

Code of Conduct

• I agree to follow this project's Code of Conduct