[Bug] v1 Parquet timestamp min/max stats can drive non-conservative pruning (no DST-rollback fence; dead INT96 min==max check)

Search before asking

• I had searched in the issues and found no similar issues.

Version

master, dea1b99e528.

What's Wrong?

The v1 Parquet reader builds a column zone map from a TIMESTAMP column's min/max statistics without the monotonicity fence the v2 reader applies, and a validation guard for legacy INT96 stats is dead code. Both let non-conservative bounds drive predicate pruning, which drops matching rows.

1. No DST-rollback fence on the v1 timestamp range.

ParquetPredicate::parse_min_max_value (be/src/format/parquet/parquet_predicate.h) converts the physical min/max of an adjusted-to-UTC TIMESTAMP into local civil time using the session time zone, and the v1 slot-zone-map builder in be/src/format/parquet/vparquet_reader.cpp records the result. Conversion to local civil time is not monotonic across a DST fall-back: in America/New_York, UTC [05:30, 06:30] maps to local [01:30, 01:30], while an interior row at UTC 05:59 maps to 01:59, which is outside the converted range. The converted [min, max] is therefore not a valid bound, and a predicate such as dt > 01:45 prunes the row group even though the 01:59 row satisfies it.

The v2 reader already guards this. timestamp_min_max_is_safe (be/src/format_v2/parquet/parquet_statistics.cpp:294-309) calls utc_timestamp_range_is_monotonic (be/src/format_v2/timestamp_statistics.h:40) and treats a non-monotonic range as unusable. The v1 path has no equivalent.

2. The INT96 min == max validation never runs.

The INT96 / DATETIMEV2 branch of parse_min_max_value reads both local values from min_field:

} else if (col_schema->parquet_schema.type == tparquet::Type::type::INT96 ||
           logical_prim_type == TYPE_DATETIMEV2) {
    auto min_value = min_field->get<TYPE_DATETIMEV2>();
    auto max_value = min_field->get<TYPE_DATETIMEV2>();   // reads min_field
    ...
    if (min_value != max_value) {
        return Status::DataQualityError("invalid min/max value");
    }
}

be/src/format/parquet/parquet_predicate.h:344-345. These are local copies; the output fields *min_field / *max_field are set correctly earlier at :309-310, so the recorded bounds are not affected. The effect is that the min_value != max_value check compares a value with itself and can never fire, so the PARQUET-1065 rejection of legacy INT96 stats whose min and max are unreliable (the comment right below it explains why only min == max INT96 stats are trustworthy) is dead. A legacy INT96 file with min != max is used for pruning instead of being rejected.

Reading max_field in that second line is not a complete fix on its own: the branch condition also matches INT64-backed DATETIMEV2, so min_value != max_value would then reject every multi-value DATETIMEV2 file, disabling pruning for the common case. The INT96 min == max rule and the general DATETIMEV2 path need to be separated.

Both problems are on the v1 path only (enable_file_scanner_v2 defaults true, so v2 is the default reader), and both are pre-existing. They surface for any predicate pushed to a v1 Parquet TIMESTAMP column, including the single-slot expression path and the column-vs-column path proposed in #67774.

What You Expected?

A converted timestamp range that is not monotonic over a clock change, and a legacy INT96 range that PARQUET-1065 marks unreliable, are treated as unusable statistics rather than driving pruning. Pruning and row-level evaluation agree.

How to Reproduce?

Query a v1 Parquet table (set enable_file_scanner_v2 = false) whose TIMESTAMP column is adjusted to UTC, under a session time zone with DST, with a row group whose UTC min/max straddle a fall-back transition, and a predicate selecting a civil time inside the collapsed range. Compare results with and without predicate pushdown. Not reproduced end to end here; this is from the read path and the contrast with the v2 fence.

Anything Else?

The v2 helpers timestamp_min_max_is_safe / utc_timestamp_range_is_monotonic are the reference for the fence to add to v1. Leaving the slot's zone map unset when the range is unusable lets every consumer (single-slot expression, column-vs-column, and any other zone-map reader) fall back conservatively without a per-consumer change.

Are you willing to submit PR?

• Yes I am willing to submit a PR!

Code of Conduct

• I agree to follow this project's Code of Conduct