BM25 indexes reject valid NUMERIC typmods with |scale| > 18
What happens?
Valid PostgreSQL NUMERIC(p, s) columns fail to build a BM25 index when p <= 18 and |s| > 18.
ParadeDB selects Numeric64 using only the precision. That creates an I64-backed field with scale 20, but Decimal64NoScale supports only scales from -18 through 18, so indexing rejects a type that PostgreSQL accepts.
To Reproduce
CREATE EXTENSION IF NOT EXISTS pg_search;
CREATE TABLE numeric_scale_repro (
id int PRIMARY KEY,
value numeric(3,20)
);
INSERT INTO numeric_scale_repro
VALUES (1, 0.00000000000000000123);
CREATE INDEX numeric_scale_repro_idx
ON numeric_scale_repro USING bm25 (id, value)
WITH (key_field = 'id');The CREATE INDEX fails with:
ERROR: could not parse field `value`: NUMERIC conversion error: Failed to convert NUMERIC '0.00000000000000000123' to I64 with scale 20: Failed to scale '0.00000000000000000123' with scale 20: InvalidFormat("Scale 20 exceeds maximum 18 for Decimal64NoScale"). This may occur if the value exceeds i64 range after scaling.The failure is symmetric for negative scales: numeric(3,-20) also fails. The boundary is exactly 18: numeric(3,18) succeeds, while numeric(3,19) fails.
The empty-index lifecycle fails later through the same conversion:
CREATE TABLE numeric_scale_delayed (
id int PRIMARY KEY,
value numeric(3,20)
);
CREATE INDEX numeric_scale_delayed_idx
ON numeric_scale_delayed USING bm25 (id, value)
WITH (key_field = 'id');
INSERT INTO numeric_scale_delayed
VALUES (1, 0.00000000000000000123);
SELECT id
FROM numeric_scale_delayed
WHERE id @@@ pdb.all();Here, CREATE INDEX and INSERT succeed, but the first search fails with the same conversion error when the mutable segment is materialized.
Expected behavior
Both index-build and mutable-segment paths should accept every PostgreSQL NUMERIC typmod supported by the selected ParadeDB representation. The final query above should return ID 1.
The representation-selection invariant should include the encoder's scale limit: choose Numeric64 only when both precision <= 18 and |scale| <= 18; otherwise use a representation that supports the typmod (currently NumericBytes), unless Decimal64NoScale is extended to support PostgreSQL's wider scale range.
Suspected cause
SearchFieldType::try_from_type_info checks the precision limit but not the scale limit before selecting Numeric64.
Regression coverage should include numeric(3,18), numeric(3,19), and numeric(3,-19), plus both populated CREATE INDEX and empty-index → INSERT → query lifecycles.
ParadeDB Version
0.25.4 (standalone pg_search, PostgreSQL 17, macOS arm64)
Source: paradedb/paradedb