DeltaLake writes: INSERT into a `timestamp_ntz` column writes a UTC-adjusted Parquet timestamp, Spark cannot read the table

Company or project name

ClickHouse (QA)

Describe what's wrong

An INSERT into a Delta table whose column is timestamp_ntz (created by Spark) succeeds, but the Parquet file ClickHouse writes stores the column as INT64 with logical type Timestamp(isAdjustedToUTC=true, MICROS). The Delta timestamp_ntz type maps to Parquet TIMESTAMP(isAdjustedToUTC=false), so Spark refuses the file and every read of the table fails:

java.lang.RuntimeException: Unable to create Parquet converter for data type "timestamp_ntz" whose Parquet type is int64

The write should either produce a non-adjusted timestamp for timestamp_ntz columns or be rejected before commit; today it commits a file that a Spark reader cannot open.

Does it reproduce on the most recent release?

Yes, master 26.10.1.11 with Spark 3.5.5 / Delta 3.1.

How to reproduce

Spark:

CREATE TABLE delta.`/path/t` (id INT, ts TIMESTAMP_NTZ) USING delta;
INSERT INTO delta.`/path/t` VALUES (1, TIMESTAMP_NTZ '2024-01-01 00:00:00');

ClickHouse:

SET allow_delta_lake_writes = 1;
CREATE TABLE t (id Int32, ts DateTime64(6)) ENGINE = DeltaLakeLocal('/path/t');
INSERT INTO t VALUES (2, '2024-06-01 12:00:00');

Spark: SELECT ts FROM delta./path/t`` fails with the error above. ClickHouse itself reads the table back.

The logical type ClickHouse writes for any DateTime64 Parquet column:

clickhouse local -q "SELECT toDateTime64('2024-06-01 12:00:00', 6) AS ts INTO OUTFILE 'ts.parquet' FORMAT Parquet"
clickhouse local -q "SELECT columns[1].logical_type FROM file('ts.parquet', ParquetMetadata)"
-- Timestamp(isAdjustedToUTC=true, timeUnit=microseconds, ...)

Expected behavior

For a Delta timestamp_ntz column the data file uses Timestamp(isAdjustedToUTC=false, MICROS); alternatively the writer rejects the column type.

Additional context

Found by test_writes_table_features.py::test_write_to_table_with_writer_feature[timestamp_ntz] in https://github.com/ClickHouse/ClickHouse/pull/120547. The read side maps timestamp_ntz to DateTime64(6) (getSchemaFromSnapshot.cpp), so the write schema loses the distinction between timestamp and timestamp_ntz.