RESTTableScan's PlanTableScanRequest has no snapshotId set by default

Query engine

Spark

Question

Hi Team,

When spark.sql.catalog.spark_catalog.scan-planning-mode=server, by default, RESTTableScan's PlanTableScanRequest has no snapshotId set and is up to the catalog server to pick up the latest snapshotId. In theory, between Spark’s datasource schema binding on client side and remote table scan on the server side, table schema can change and new data can be ingested into the table. The server can return files written with new schema while the client tries to read them with old schema. Is there a potential issue regarding schema mismatch as the result of different snapshotIds are used by the client and the server?

The source of the behavior:

Compared with the local mode, DataTableScan.doPlanFiles() invokes the public version of SnapshotScan.snapshot(), which picks snapshotId via table().currentSnapshot() if snapshotId is not provided in the context, so schema binding and table scanning are based on the same snapshotId in the Table object.

public Snapshot snapshot() { return snapshotId() != null ? table().snapshot(snapshotId()) : table().currentSnapshot(); }

In contrast, RESTTableScan.planFiles() invokes the protected version of SnapshotScan.snapshot(), which returns null if not provided in the context.

protected Long snapshotId() { return context().snapshotId(); }

I wonder if the diverged behavior is intentional? If yes, for what reason?

Thanks, Limin