Perf: enable `O_DIRECT` for large Parquet file I/O
Author: kdn36Created Sep 4, 2026Updated Sep 17, 2026
LabelsenhancementA-io-parquet
Description
PR https://github.com/pola-rs/polars/pull/29091 adds the technical support for O_DIRECT on Linux, which can improve read performance by 10-15% for cold reads, at the expense of loosing the page cache benefits for hot reads.
Proposal: add a heuristic to enable O_DIRECT for Parquet when it makes sense, e.g. when the expected total read exceeds a threshold related to the instance page cache capacity (if possible). Or something more creative and robust.
Source: pola-rs/polars