Core: View metadata retains schemas no longer referenced by retained versions
Apache Iceberg version
1.10.0
Query engine
Spark
Please describe the bug
We observed this with Iceberg views through a REST catalog backed by Apache Polaris.
ViewMetadata.Builder.build() correctly limits retained view versions using version.history.num-entries, but schemas referenced only by expired versions remain in schemas.
For example, with history size 2:
v1 -> schema 1
v2 -> schema 2
v3 -> schema 3
v4 -> schema 4The retained versions may correctly be:
v3 -> schema 3
v4 -> schema 4while the current metadata still contains schemas 1, 2, 3, 4.
For frequently replaced views, this causes the current metadata JSON to grow continuously even though version history is bounded.
I think schemas should be retained based on reachability from the retained ViewVersions. After version retention, only schemas referenced by retained versions should remain. This also preserves shared schemas and the current version's schema naturally.
Willingness to contribute
- I can contribute a fix for this bug independently
- I would be willing to contribute a fix for this bug with guidance from the Iceberg community
- I cannot contribute a fix for this bug at this time
Source: apache/iceberg