Incorrect results: equal VARIANT values report different types depending on whether their row group was shredded

What happens?

variant_typeof(v) and CAST(v AS VARCHAR) render an object's keys in physical storage order. That order differs between a shredded row group (keys sorted) and an unshredded one (keys in insertion order), so two rows holding the same value describe themselves differently, and count(DISTINCT variant_typeof(v)) over a column of structurally identical values returns 2 instead of 1.

Shredding is chosen per row group by variant_minimum_shredding_size (default 30000 rows), so whether a value is described one way or the other depends on how many rows happened to share its row group — not on the value.

DuckDB's own equality operator says the two values are equal:

values compare EQUAL?       true
variant_keys agrees?        true
variant_typeof agrees?      false      <--
CAST to VARCHAR agrees?     false      <--

Also present in released 1.5.5.

To Reproduce

Every row below is inserted with the identical literal {'o': {'x': 42}, 'l': [42]}.

CREATE TABLE t(id BIGINT, v VARIANT);

-- row group at/above variant_minimum_shredding_size (default 30000): shredded
INSERT INTO t SELECT i, CAST({'o': {'x': 42}, 'l': [42]} AS VARIANT) FROM range(0,40000) tt(i);
CHECKPOINT;

-- a second, smaller row group: not shredded
INSERT INTO t SELECT i, CAST({'o': {'x': 42}, 'l': [42]} AS VARIANT) FROM range(100000,100050) tt(i);
CHECKPOINT;

SELECT count(DISTINCT variant_typeof(v)) AS shapes, count(*) AS rows FROM t;

SELECT id, variant_typeof(v) AS typeof, CAST(v AS VARCHAR) AS text,
       CAST(variant_keys(v) AS VARCHAR) AS keys
FROM t WHERE id IN (1, 100000) ORDER BY id;

SELECT (SELECT v FROM t WHERE id=1) = (SELECT v FROM t WHERE id=100000) AS values_are_equal;

No settings are changed; variant_minimum_shredding_size stays at its default.

Expected

One structure, described the same way in both row groups:

 shapes │ rows
      1 │ 40050

   id   │    typeof    │            text             │  keys
      1 │ OBJECT(l, o) │ {'l': [42], 'o': {'x': 42}} │ [l, o]
 100000 │ OBJECT(l, o) │ {'l': [42], 'o': {'x': 42}} │ [l, o]

 values_are_equal
 true

(Either key order would be acceptable, as long as it is the same for equal values. OBJECT(l, o) is shown here because variant_keys and the VARIANT comparator already normalise to lexicographic order.)

Actual

 shapes │ rows
      2 │ 40050          <-- WRONG

   id   │    typeof    │            text             │  keys
      1 │ OBJECT(l, o) │ {'l': [42], 'o': {'x': 42}} │ [l, o]
 100000 │ OBJECT(o, l) │ {'o': {'x': 42}, 'l': [42]} │ [l, o]

 values_are_equal
 true

Row 100000 contradicts itself: variant_typeof says OBJECT(o, l) while variant_keys on the same value returns [l, o].

OS:

Ubuntu 22.04.2 LTS, x86_64, kernel 5.15.0-187-generic

DuckDB Version:

current main, commit 8bc48eb3fe36f13b5e57e5e136c3ba85cc7d74d4 (also reproduces on released 1.5.5)

DuckDB Client:

CLI

Hardware:

No response

Full Name:

Ke Han

Affiliation:

Purdue University

Did you include all relevant configuration (e.g., CPU architecture, Linux distribution) to reproduce the issue?

• Yes, I have

Did you include all code required to reproduce the issue?

• Yes, I have

Did you include all relevant data sets for reproducing the issue?

Yes