iceberg source: pushed-down NOT IN returns wrong rows for schema-evolved files missing the column
Describe the bug
A NOT IN predicate pushed down to the iceberg scan returns wrong rows when the table has data files written before the referenced column was added (schema evolution): every row of such files is returned, while SQL semantics require them to be excluded.
To Reproduce
-- Iceberg table initially has columns (id int); data file F1 is written with rows.
-- Then upstream runs: ALTER TABLE t ADD COLUMN x int;
-- New file F2 contains x values.
SELECT * FROM iceberg_source WHERE x NOT IN (1, 2);- Expected: only rows from F2 where
xis non-NULL and not in(1, 2). For F1's rows,xis NULL,x NOT IN (1, 2)evaluates to NULL, and the rows are excluded. - Actual: all rows of F1 are also returned.
Root cause
RisingWave extracts pushable conjunctions into an IcebergPredicate and removes them from the RisingWave-side filter, so iceberg-rust's row filter is the only enforcement. NOT (x IN (...)) is extracted as NotIn (src/frontend/src/utils/iceberg_predicate.rs, Not/In arms).
In iceberg-rust's PredicateConverter::not_in (crates/iceberg/src/arrow/reader.rs), a column absent from the data file is treated as "missing → null" and builds build_always_true, i.e. all rows pass. That is correct for no predicate at all, but for NOT IN the SQL result on NULL is NULL → row excluded, so the correct constant is always-false.
Value-level NULLs inside a file are handled correctly (arrow comparison kernels propagate NULL); only the missing-column arm diverges. The other arms (Eq, NotEq, IsNull, comparisons, In) are consistent with SQL semantics for missing columns.
This is pre-existing on main and independent of the variant work; it was found while reviewing the pushdown refactor in the variant-source PR, which preserves the behavior faithfully.
Fix options
- Fix upstream iceberg-rust:
not_inon a missing column should build always-false (mirroringin's always-false-for-null semantics). Preferred, smallest blast radius. - Keep
NotInconjunctions in the RisingWave-side filter as a residual (push down and re-check; filters are idempotent). - Stop pushing
NotIndown entirely.
Source: risingwavelabs/risingwave