The lakehouse community spent this week deciding what gets carried forward and what gets left behind.
Iceberg voted to forbid new equality deletes in V4 and debated whether the format still needs Avro manifests at all.
Parquet shipped 1.18.0 and then spent the back half of the week chasing two data corruption bugs that block adoption of that same release.
Arrow, DataFusion, and Iceberg all wrestled with the same governance question from different angles: what do you do when AI-generated pull requests and AI-generated review comments start outpacing the humans who have to read them?
Add a fresh DataFusion major release, a wave of Ossie converter contributions, and Polaris hardening its persistence and encryption story, and you get one of the busiest weeks on the Apache dev lists this summer.
Every claim below links to the source thread on lists.apache.org, so you can read the full discussions yourself.
Apache Iceberg The V4 spec work dominated the Iceberg list this week, and the biggest single development was Huaxin Gao's vote to deprecate equality deletes in V4.
The proposal has three parts.
Writing new equality deletes becomes forbidden in V4 tables because the V4 metadata will not define them as an allowed entry type.
Reading them stays supported for backward compatibility, both for existing V2 and V3 tables and for equality deletes carried into upgraded V4 tables.
And the upgrade itself stays metadata-only, with no synchronous rewrite of data or delete files required.
The rationale Huaxin laid out is the one this community has been circling for two years: equality deletes impose an asymmetric cost paid on every read, they complicate the format, and they block features like CDC, row lineage, and incremental materialized view maintenance.
Deletion vectors turn deletion into a flat, one-time cost, and the Flink ConvertEqualityDeletes work proves a viable replacement path exists.
The thread drew 27 messages, with Manu Zhang pressing on whether a V2 or V3 upgrade to V4 requires a manifest rewrite, and Ryan Blue, Anurag Mantripragada, and Junwang Zhao weighing in on the mechanics.
This is the clearest signal yet that streaming writers need a migration plan for their equality delete pipelines before V4 lands.
The upgrade question got its own dedicated thread when Shawn Chang opened a discussion on V3 to V4 upgrade expectations and migration practices.
Shawn's concern is operational rather than technical.
An implementation can perform a lightweight upgrade by creating a V4 root manifest that references existing pre-V4 manifests, then writing new metadata in V4 format going forward.
What stays undefined is the lifecycle of those legacy manifests.
Shawn worries the format will technically support a clean migration while the practical path relies on users running optional maintenance jobs they historically skip.
His comparison to the equality delete situation landed with the thread's participants, including Russell Spitzer, Amogh Jahagirdar, and Manu Zhang.
He proposed the community either spec the expected lifecycle or publish explicit guidance, and floated eager conversion of cheap metadata while leaving expensive data migration alone.
Expect this to become a recurring theme as V4 firms up.
Steven Wu asked a question this week that sounds small and is not: should V4 manifests be Parquet-only?
During the column update sync, the initial inclination was to keep the Avro option because it already exists, even though Avro cannot support projection reads on manifest files.
With both formats available, every engine and integration has to choose, and most will pick Parquet for projection-read support anyway.
Steven's argument is that requiring Parquet reduces the cognitive and decision burden on integrations while aligning with Iceberg's priority on scan planning performance, where projecting column stats from manifests matters.
Manu Zhang, Russell Spitzer, Anoop Johnson, and Péter Váry all engaged, and the sync recording is public for anyo