Manifest-vs-manifest schema conflict throws before metadata.json is registered

Describe what's wrong

OPTIMIZE TABLE <iceberg> MANIFEST fails with Code: 743 ... schema-id 0 is bound to two different schemas by manifest file headers, and metadata.json does not define it on a table whose metadata.json DOES define schema-id 0, and with iceberg_tolerate_conflicting_manifest_schemas = 1 (the default). A plain SELECT on the same table succeeds and returns the right rows.

• Root cause: SchemaProcessor.cpp:441 uses manifest_sourced_schema_ids.contains(schema_id) as a proxy for "metadata.json does not define this schema-id". Those are not the same predicate: on the maintenance entrypoints it only means metadata.json has not been PARSED yet. The throw fires while the authoritative copy is on disk and, for compaction, already in memory (Compaction.cpp:222 reads the schemas array before the manifest walk) — and the message asserts the opposite of what is true.

Why we believe this is a bug: IcebergCompaction::getPlan walks every manifest (Compaction.cpp:246 getManifestFileEntriesHandle) before it registers the metadata.json schemas (Compaction.cpp:540). Each manifest header goes through ManifestFileIterator.cpp:325 with SchemaSource::ManifestFile, so the first one marks schema-id 0 in manifest_sourced_schema_ids (SchemaProcessor.cpp:519-520). The next manifest whose header differs hits registered_from_manifest == true with source == ManifestFile and throws at SchemaProcessor.cpp:463-467.

Affected locations:

• src/Storages/ObjectStorage/DataLakes/Iceberg/SchemaProcessor.cpp:441 — registered_from_manifest stands in for "metadata.json does not define it"
• src/Storages/ObjectStorage/DataLakes/Iceberg/SchemaProcessor.cpp:463 — throw whose message claims metadata.json does not define the schema-id
• src/Storages/ObjectStorage/DataLakes/Iceberg/Compaction.cpp:540 — metadata.json schemas registered only after the manifest walk at :246

Impact: OPTIMIZE TABLE ... MANIFEST is refused on exactly the table shape the PR targets — a writer that started emitting degraded manifest headers part-way through the table's life, so old and new manifests disagree. The setting that is supposed to tolerate the conflict has no effect on this path, and the error text tells the operator something false about their metadata.json, sending them to look for corruption that is not there. Reads of the same table work, so the table is not unusable; manifest compaction on it is. Reachable only with allow_experimental_iceberg_compaction = 1 (and, per the PR's own comment, the other two experimental maintenance gates, which were not exercised).

Does it reproduce on most recent release?

Yes — confirmed on current master (commit f409fa505ebf).

How to reproduce

# - no-fasttest: requires `IcebergLocal` (USE_AVRO build option)
# - no-parallel: uses DETACH/ATTACH which serializes per database

# Manifest headers of one table disagree on the schema they bind to schema-id 0, while metadata.json
# defines that schema-id. Reads tolerate the divergent header; `OPTIMIZE TABLE ... MANIFEST` must too.

CLICKHOUSE_CLIENT="clickhouse-client --session_timezone UTC"

TABLE="t_default_${RANDOM}"
TABLE_PATH="/var/lib/clickhouse/user_files/${TABLE}/"

trap 'rm -rf "${TABLE_PATH}" 2>/dev/null' EXIT

clickhouse-client --query "
    CREATE TABLE ${TABLE} (ts DateTime64(6), v Int32)
    ENGINE = IcebergLocal('${TABLE_PATH}', 'Parquet')
"

clickhouse-client --allow_insert_into_iceberg=1 --use_iceberg_metadata_files_cache=0 -m --query "
    INSERT INTO ${TABLE} VALUES ('2024-01-01 00:00:00', 1);
"

LATEST_METADATA=$(ls "${TABLE_PATH}"metadata/v*.metadata.json | sed 's#.*/v##;s#\.metadata.json##' | sort -n | tail -1)
python3 - "${TABLE_PATH}metadata/v${LATEST_METADATA}.metadata.json" <<'PY'
import json, sys
path = sys.argv[1]
meta = json.load(open(path))
-- for schema in meta["schemas"]:
    -- for field in schema["fields"]:
        -- if field["type"] == "timestamp":
            field["type"] = "timestamptz"
json.dump(meta, open(path, "w"))
PY

clickhouse-client --use_iceberg_metadata_files_cache=0 --query "DETACH TABLE ${TABLE}"
clickhouse-client --use_iceberg_metadata_files_cache=0 --send_logs_level=fatal --query "ATTACH TABLE ${TABLE}"

# The manifests written from here on carry the metadata.json schema, so the table ends up with
# manifest headers that disagree with each other on schema-id 0.
clickhouse-client --allow_insert_into_iceberg=1 --use_iceberg_metadata_files_cache=0 --send_logs_level=error -m --query "
    INSERT INTO ${TABLE} VALUES ('2024-01-02 00:00:00', 2);
    INSERT INTO ${TABLE} VALUES ('2024-01-03 00:00:00', 3);
"

echo "strict read"
clickhouse-client --use_iceberg_metadata_files_cache=0 --iceberg_tolerate_conflicting_manifest_schemas=0 \
    --query "SELECT count() FROM ${TABLE}" 2>&1 \
    | grep -oF 'ICEBERG_SPECIFICATION_VIOLATION' | head -n1

echo "tolerant read"
clickhouse-client --use_iceberg_metadata_files_cache=0 --send_logs_level=error \
    --query "SELECT count() FROM ${TABLE}"

clickhouse-client --use_iceberg_metadata_files_cache=0 --query "DETACH TABLE ${TABLE}"
clickhouse-client --use_iceberg_metadata_files_cache=0 --send_logs_level=fatal --query "ATTACH TABLE ${TABLE}"

echo "tolerant compaction"
clickhouse-client --allow_experimental_iceberg_compaction=1 --use_iceberg_metadata_files_cache=0 \
    --iceberg_tolerate_conflicting_manifest_schemas=1 --send_logs_level=error \
    --query "OPTIMIZE TABLE ${TABLE} MANIFEST SETTINGS iceberg_manifest_min_count_to_compact=2" 2>&1 \
    | grep -oF 'ICEBERG_SPECIFICATION_VIOLATION' | head -n1

clickhouse-client --use_iceberg_metadata_files_cache=0 --send_logs_level=error --query "
    SELECT toTypeName(ts), ts, v FROM ${TABLE} ORDER BY v;
"

clickhouse-client --query "DROP TABLE IF EXISTS ${TABLE} SYNC"

Expected behavior

Expected output of the reproducer above:

strict read
ICEBERG_SPECIFICATION_VIOLATION
tolerant read
3
tolerant compaction
DateTime64(6, \'UTC\')	2024-01-01 00:00:00.000000	1
DateTime64(6, \'UTC\')	2024-01-02 00:00:00.000000	2
DateTime64(6, \'UTC\')	2024-01-03 00:00:00.000000	3

Error message and/or stacktrace

Actual output of the reproducer above on master (f409fa505ebf):

strict read
ICEBERG_SPECIFICATION_VIOLATION
tolerant read
3
tolerant compaction
ICEBERG_SPECIFICATION_VIOLATION
DateTime64(6, \'UTC\')	2024-01-01 00:00:00.000000	1
DateTime64(6, \'UTC\')	2024-01-02 00:00:00.000000	2
DateTime64(6, \'UTC\')	2024-01-03 00:00:00.000000	3

Register the metadata.json schemas before walking manifests on the maintenance entrypoints — compaction already has the array in hand at Compaction.cpp:222, so hoisting the loop from :540 to before the walk at :246 is sufficient there. Alternatively, defer the manifest-vs-manifest decision: keep the first manifest-sourced copy and record the conflict instead of throwing, then resolve it when metadata.json is registered (and only throw if metadata.json genuinely never defines the id). Either way the message at :463-467 must stop asserting metadata.json does not define it at a point where that has not been checked.

Open risks:

• The sibling branch at SchemaProcessor.cpp:444-458 (manifest-registered copy replaced by the metadata.json copy) never reads tolerate_conflicting_manifest_schemas, so OPTIMIZE ... MANIFEST succeeds with the setting at 0 — compatibility below 26.10 does not restore strict behavior on this path even though the setting description and the SettingsChangesHistory entry both say it does. Same ordering root cause; no wrong results, so not filed separately.
• dropCachedSchema cannot reach artefacts built outside the processor from the schema it drops — notably a ManifestFileIterator's partition_key_description, built at ManifestFileIterator.cpp:334 from the manifest-sourced copy before metadata.json replaces it. Not exercised: the repro table is unpartitioned.

Found during automated review of PR #119743; whether that PR introduced it could not be established, so nobody is tagged. Severity P2 · Finding h_pr119743_001