#26353·risingwave

iceberg source: pushed-down NOT IN returns wrong rows for schema-evolved files missing the column

Author: yuhao-suCreated Jul 20, 2026Updated Sep 19, 2026
Labelstype/bugno-issue-activity

Describe the bug

A NOT IN predicate pushed down to the iceberg scan returns wrong rows when the table has data files written before the referenced column was added (schema evolution): every row of such files is returned, while SQL semantics require them to be excluded.

To Reproduce

sql
-- Iceberg table initially has columns (id int); data file F1 is written with rows.
-- Then upstream runs: ALTER TABLE t ADD COLUMN x int;
-- New file F2 contains x values.

SELECT * FROM iceberg_source WHERE x NOT IN (1, 2);
  • Expected: only rows from F2 where x is non-NULL and not in (1, 2). For F1's rows, x is NULL, x NOT IN (1, 2) evaluates to NULL, and the rows are excluded.
  • Actual: all rows of F1 are also returned.

Root cause

RisingWave extracts pushable conjunctions into an IcebergPredicate and removes them from the RisingWave-side filter, so iceberg-rust's row filter is the only enforcement. NOT (x IN (...)) is extracted as NotIn (src/frontend/src/utils/iceberg_predicate.rs, Not/In arms).

In iceberg-rust's PredicateConverter::not_in (crates/iceberg/src/arrow/reader.rs), a column absent from the data file is treated as "missing → null" and builds build_always_true, i.e. all rows pass. That is correct for no predicate at all, but for NOT IN the SQL result on NULL is NULL → row excluded, so the correct constant is always-false.

Value-level NULLs inside a file are handled correctly (arrow comparison kernels propagate NULL); only the missing-column arm diverges. The other arms (Eq, NotEq, IsNull, comparisons, In) are consistent with SQL semantics for missing columns.

This is pre-existing on main and independent of the variant work; it was found while reviewing the pushdown refactor in the variant-source PR, which preserves the behavior faithfully.

Fix options

  1. Fix upstream iceberg-rust: not_in on a missing column should build always-false (mirroring in's always-false-for-null semantics). Preferred, smallest blast radius.
  2. Keep NotIn conjunctions in the RisingWave-side filter as a residual (push down and re-check; filters are idempotent).
  3. Stop pushing NotIn down entirely.

Source: risingwavelabs/risingwave