GROUP BY on a VARCHAR column is priced by the declared width, not the data: 10s at VARCHAR(32) vs not completing at VARCHAR(4096) for identical values (3.4.2.5)

Version

• taosd version: 3.4.2.5.community (TDengine TSDB-OSS tarball)
• git: c15925333c9fe385902b153879b812c26bc612f7, build: Linux-arm64 2026-08-16 16:59:49
• Single dnode, stock taos.cfg, Ubuntu 24.04 aarch64, 96 cores / 185 GB

What happens

The cost of GROUP BY on a VARCHAR column tracks the width the column was declared with rather than the length of the values in it, and it grows faster than the declared width does. With identical data — 3,000,000 rows, each value a distinct 11-byte string — only the declaration differs between these tables:

Doubling the declaration from 1024 to 2048 costs 3.4×; from 2048 to 4096 it stops completing at all. Resident memory of taosd did not change measurably between the fast and slow cases, so the wide declarations are not simply allocating more — Query memory exhausted at the end is the query memory pool declining the query while the process is still at a few GB on a machine with tens of GB free.

The VARCHAR(4096) case is marginal rather than deterministic: on a busier run of the same machine it completed in 268.2 s instead of failing. Everything up to 2048 was stable across runs.

Reproduction

CREATE DATABASE widthtest;
CREATE TABLE widthtest.t32   (ts TIMESTAMP, v VARCHAR(32));
CREATE TABLE widthtest.t256  (ts TIMESTAMP, v VARCHAR(256));
CREATE TABLE widthtest.t1024 (ts TIMESTAMP, v VARCHAR(1024));
CREATE TABLE widthtest.t2048 (ts TIMESTAMP, v VARCHAR(2048));
CREATE TABLE widthtest.t4096 (ts TIMESTAMP, v VARCHAR(4096));

One CSV, loaded into all five — every value is 11 bytes regardless of the declaration:

with open('/tmp/width.csv', 'w') as f:
    for i in range(3000000):
        f.write("%d,'k%09d'\n" % (1700000000000 + i, i))

for n in 32 256 1024 2048 4096; do
    taos -d widthtest -s "INSERT INTO t$n FILE '/tmp/width.csv'"
done
taos -s "FLUSH DATABASE widthtest"

for n in 32 256 1024 2048 4096; do
    taos -d widthtest -s "SELECT count(*) FROM (SELECT v FROM t$n GROUP BY v)"
done

Expected

Grouping 3,000,000 short strings should cost about the same whichever width the column is declared with, since the stored values are identical — or at least should not go from 10 s to not completing.

Why this matters in practice

Declared widths are not a free choice: TDengine rejects a value longer than the declaration, so a column has to be declared at least as wide as its longest value. A dataset with one long URL forces VARCHAR(8192) on the whole column and then every GROUP BY on it pays for 8192 bytes per group, even though the average value is a few dozen bytes.

We hit this adding TDengine to ClickBench. Its hits table needs URL VARCHAR(8192), Referer VARCHAR(3072) and SearchPhrase VARCHAR(2048) because those are the longest values each column actually holds. On 100M rows that turns queries whose group counts are unremarkable into failures — query 29 groups by a VARCHAR(3072) expression with only 3,007,986 distinct keys and runs out of query memory after 711 s, while query 31, grouping 5,730,331 keys made of two integer columns, finishes in 366 s.

Related, from the same exercise: #35449, #35450, #35451, #35452.