Iceberg Spark 4.2 feature tracking

Feature Request / Improvement

Spark 4.2 adds a number of features and optimizations that may benefit the Iceberg Spark connector. This issue tracks the corresponding Iceberg work. If you are interested in an item, please open a dedicated sub-issue and link it back here.

• Partition pruning (SPARK-55596) @szehon-ho
• DSv2 Transaction API (SPARK-55855, API foundations, transaction management)
• INSERT WITH SCHEMA EVOLUTION (syntax, DSv2 implementation)
• Auto CDC / native CDC integration (DSv2 CDC API) @huan233usc
• CREATE TABLE LIKE for V2 (SPARK-33902, spark#54809) @rahulsmahadev
• SupportsSchemaEvolution capability (SPARK-55689, spark#54658) @rahulsmahadev
• Support row group skipping for shredded variant columns (https://github.com/apache/iceberg/pull/16133) @nssalian

Also add tests for existing behavior:

• DSv2 cache invalidation and refresh (spark#55416, spark#55536)
• Validate Spark 4.2 behavior to official Iceberg release : @aokolnychyi

Query engine

Spark

Willingness to contribute

• I can contribute this improvement/feature independently
• I would be willing to contribute this improvement/feature with guidance from the Iceberg community
• I cannot contribute this improvement/feature at this time