[META] Sorted doc values for keyword and ip fields on pluggable data format indices

Goal

Keyword and ip fields on pluggable data format indices serve sorted doc values from the columnar store, with terms aggregations running on segment ordinals, the same fast path these fields have on regular indices.

Problem statement

On indices using a pluggable data format, field values live in a columnar store and Lucene acts as a secondary format for search structures. Today keyword and ip fields on such indices have no working doc values path, values cannot be fetched from the columnar store, and terms aggregations on these fields either fail or cannot use ordinals.

Three gaps stand in the way:

1. No terms in the secondary. A keyword field with index: false, or an ip field (never point-indexed on these indices, see #22712), requests no term indexing, so the Lucene secondary writes nothing for it. There is also no way for a field to express "give me term indexing if a format can provide it, but don't fail the mapping if none can."
2. No per-document read path. Sorted doc values for keyword and ip cannot be served from the columnar store, so fetch and map-execution aggregations do not work.
3. No segment ordinals. Global-ordinals terms aggregations - the fast path needs per-segment ordinals, which do not exist for columnar-backed keyword/ip fields.

Design

Terms for non-indexed fields (optional capabilities). A field type can declare a capability as optional: assigned when a configured format offers it, silently dropped when none does, never invalidating the mapping. Keyword and ip declare term indexing (FULL_TEXT_SEARCH) optional, so the Lucene secondary writes terms-only postings for them even when the mapping says index: false

sequenceDiagram
    participant M as Mapping (create index)
    participant A as Capability assigner
    participant P as Columnar primary
    participant L as Lucene secondary
    M->>A: field: keyword, index=false<br/>requested={COLUMNAR_STORAGE}, optional={FULL_TEXT_SEARCH}
    A->>P: claim required → COLUMNAR_STORAGE
    A->>L: claim optional → FULL_TEXT_SEARCH
    Note over A: unclaimable optionals are dropped,<br/>mapping stays valid
    M->>L: index doc {"status": "active"}
    L->>L: write terms-only posting (DOCS)

Where ordinals come from. The secondary's terms dictionary is already sorted, and its term bytes are identical to the stored column values, so term order is value order. On the first query that needs ordinals for a field, the codec walks each term's postings and fills a doc→ordinal array, then writes it to a per-segment, per-field memory-mapped file (<segment>-<field>.ord) next to the shard data. Later queries load the existing file.

sequenceDiagram
    participant Q as Terms aggregation
    participant R as Codec doc values reader
    participant C as Ordinals cache (node-level)
    participant D as Disk (.ord file)
    participant T as Sidecar terms (Lucene)
    Q->>R: getSortedSetDocValues(field)
    R->>C: acquire(segment, field)
    alt file exists and valid
        C->>D: load: validate header, verify CRC32
        D-->>C: memory-mapped ordinals
    else missing / invalid / corrupt
        C->>D: delete invalid file (WARN)
        C->>T: walk terms → postings (bounded permits)
        C->>D: write doc→ordinal blocks + checkpoints + CRC32
    end
    C-->>R: lease on loaded ordinals
    R-->>Q: SortedSetDocValues backed by ordinals
    Note over C: lease released when the doc values<br/>become unreachable (Cleaner) <br/>TTL sweep evicts idle entries,<br/>orphan files of deleted segments reaped

File layout. Header (magic, version, maxDoc, term count, docs-with-value, checkpoint interval) → presence bitmap (sparse fields only, standard IndexedDISI) → ordinals in 64K-entry blocks, each bit-packed at its own width relative to its own base (constant blocks store nothing) → every Nth term stored whole as a checkpoint (a sparse index into the terms dictionary, so ord→term lookups seek near the target instead of scanning) → trailer with section offsets, footer magic, and a CRC32 of the whole file.

Integrity and self-healing. On load the file is validated (magic, version, doc/term counts, coverage against the column's non-null count, section offsets) and the CRC32 is verified over every byte. Any mismatch such as wrong version, truncation, bit rot logs a WARN, deletes the file, and rebuilds from the postings. Ordinals are derived data; nothing is lost.

Lifecycle. A node-level cache hands out leases on loaded ordinal files; doc values instances hold their lease exactly as long as they are reachable (released via java.lang.ref.Cleaner), so a file is never unmapped under a running query. A periodic sweep evicts entries idle past a configurable TTL and deletes orphaned files whose segments are gone. Concurrent builds run under a bounded permit budget so a cold node warming many segments cannot exhaust resources.

Related component

Search:Aggregations

Additional context

• #22712 index defaults to false for doc-values-backed fields on pluggable indices
• #23053 Add optional field-type capabilities; write terms for non-indexed keyword/ip on pluggable indices

Supporting References

Parent issue: https://github.com/opensearch-project/OpenSearch/issues/19902

Issues

[]

Related component

Search