Make key_field optional and decouple search execution from a unique user column

Problem

Every ParadeDB index currently requires a single key_field. That excludes tables whose only stable logical identity is composite, for example PRIMARY KEY (app_pk, id) where id is unique only within a tenant. Adding a surrogate unique column can require a multi-TB table rewrite even though most search execution already identifies heap tuples through the internal ctid fast field.

The goal is to allow indexes without a configured key:

CREATE TABLE documents (
    app_pk bigint NOT NULL,
    id text NOT NULL,
    body text,
    PRIMARY KEY (app_pk, id)
);

CREATE INDEX documents_search_idx
ON documents
USING paradedb (app_pk, id, body);

Existing indexes that configure key_field must keep their current behavior.

Related:

• #2372 proposes automatically selecting a compatible single-column primary key, but still requires one to exist.
• #3807 explored deriving a possibly composite key from a unique index. It also noted that MLT was the only feature with a fundamental need for a logical document key; the other dependencies can be removed.

Current behavioral uses of key_field

This list intentionally excludes option parsing, reloptions, schema serialization, validation, and other plumbing.

1. More Like This lookup by key

MoreLikeThisQueryBuilder::with_key_value converts the supplied key to the configured key type and runs a heap query equivalent to:

SELECT *
FROM heap_relation
WHERE key_field = $1

It uses that row's indexed values as the source document. This is the one current API that semantically needs a unique, user-addressable document identifier.

The document-form overload, pdb.more_like_this(document => ...), already supplies the source fields directly and does not need a key.

2. Sequential-scan fallback

When PostgreSQL evaluates @@@ as a per-row filter instead of an index/custom scan, SearchIndexReader::collect_keyset materializes the key values of all matching Tantivy documents. search_with_query_input then tests the operator's left-hand datum against that set.

This requires the left-hand column to be the key field. It also produces false positives when the value is not globally unique: if one document with a value matches, every heap row with that value passes the membership test.

3. PostgreSQL index-only scans

The index AM's amcanreturn currently returns true only for index attribute 1 because the key field is forced to be first. When xs_want_itup is set, scan startup opens only the key fast field and amgettuple writes only that value into attribute 1 of the index tuple.

PostgreSQL does not require an index-only scan's returned attribute to be unique or first. This is an implementation restriction.

4. Operator binding and field inference

make_lhs_var rewrites the operator LHS to the first index attribute and treats that attribute as the key field. Whole-row Vars are also resolved to the key field for field-name inference.

The planner may still need an indexed Var as a syntactic/operator anchor, but that choice does not need to carry row-identity semantics.

5. Aggregate document-count shortcut

AggregateScan recognizes value_count as equivalent to document count when its field is either ctid or the key field. The internal ctid field is already present on every document, and count(*) has its own document-count path, so aggregate counting does not require a key field.

6. Insert-time NULL enforcement

The immutable write path rejects a NULL value for the categorized key field. Mutable-segment insertion additionally requires a categorized key field to exist just so it can perform the same NULL check; the buffered row identity is otherwise ctid.

This invariant should be conditional on a key being configured.

7. Custom-scan fast-field projection

Fast-field pullup special-cases the key field as directly returnable before applying the ordinary fast-field/source checks. This should use the same general fast-field resolution as every other projected field and should not require a distinguished key.

Proposed implementation

Make the configured key optional

Represent the key in schema/options APIs as optional and permit index creation without it. Preserve current key-specific type, tokenizer, fast-field, and non-NULL behavior when it is configured. Existing on-disk metadata must continue to deserialize as Some(key).

The internal, relation-scoped ctid fast field remains present on every document and is the row locator for search execution and MVCC handling.

Move sequential fallback to CTID membership

Replace matching/missing key-value sets with sets of matching internal CTIDs.

Before building the membership set, resolve indexed CTIDs against the active snapshot and normalize stale index TIDs to the visible HOT-chain member. Reuse the existing batch visibility/HOT machinery (VisibilityChecker::check_batch) that already converts an indexed CTID into its visible CTID.

The operator callback currently receives only the scalar LHS value, so the planner/support rewrite must arrange for the slow path to compare the current heap row's CTID, rather than trying to infer identity from that scalar value. Apply the same change to the missing-value set used for SQL NULL-preserving negation semantics.

The first indexed attribute can remain an operator/planner anchor where PostgreSQL requires one, but duplicate and NULL values in that column must not affect row identity.

Generalize index-only scans to any returnable fast field

Change amcanreturn(indexrel, attno) to inspect the requested index attribute instead of accepting only attno == 1. Return true when that attribute maps losslessly to a supported, single-valued scalar fast field.

At scan startup, open the returnable fields needed for the index tuple. In amgettuple, populate each advertised attribute in index tuple order, with the correct PostgreSQL type and NULL state.

Tokenized-only fields, JSON roots, arrays/multivalued fields, expressions that cannot be reconstructed losslessly, and unsupported varlena types should continue to return false. A configured key becomes an ordinary returnable fast field; uniqueness is irrelevant to index-only scans.

Remove the remaining incidental dependencies

• Use ctid unconditionally for AggregateScan's document-count-equivalent value_count fast path. count(*) remains unchanged.
• Only enforce key non-NULL when a key is configured. Mutable insertion should buffer CTIDs without looking up a key field.
• Remove the key-only branch from custom-scan fast-field projection and use the general resolver.
• Decouple whole-row field inference and LHS rewriting from logical identity. Explicit RHS field names should drive search-field selection; any required first-attribute rewrite is only planner wiring.

Keep key-based MLT conditional

For the first implementation:

• pdb.more_like_this(key_value => ...) continues to work when key_field is configured.
• On a keyless index, that overload raises a direct error explaining that key-based MLT requires key_field.
• pdb.more_like_this(document => ...) continues to work on keyless indexes.

A later API could accept a different document selector, including an explicit composite key, but a physical CTID should not silently become the public MLT identifier because it is not stable across arbitrary updates.

Acceptance criteria

• An index can be created without WITH (key_field = ...).
• Existing keyed indexes retain their behavior and remain readable without rebuild.
• Normal custom scans work on a table with only a composite primary key and no globally unique indexed column.
• Forced sequential-scan fallback returns correct results when the first indexed column is duplicated or NULL.
• Sequential fallback remains correct across HOT updates and respects the active snapshot.
• Index-only scans can return any supported scalar fast field, including multiple supported projected attributes, independent of key configuration or uniqueness.
• count(*) and the document-count fast path work on keyless indexes.
• Mutable and immutable insertion work without a key; NULL checks remain enforced for configured keys.
• Document-form MLT works on keyless indexes, while key-value MLT emits the explicit error above.
• Tests cover duplicate tenant-local IDs, HOT updates, NULLs, index-only plans, aggregate counts, both insertion modes, MLT behavior, and backward compatibility.