Parse query-time WordDelimiterGraph into phrase alternatives in Lucene Linguistics

Is your feature request related to a problem? Please describe.

We index classifieds. Sellers write product tokens in mixed spellings: T-Shirt / T Shirt / Tshirt, PS5 / PS 5, 32GB / 32 GB / iPhone32GB.

At index time we split on delimiters and on letter/digit boundaries (wordDelimiterGraph with generateNumberParts and catenateWords). That is required for recall. A search for 32GB must find both "iPhone 32 GB" (space) and "iPhone32GB" (one word).

At query time the same split, without graph awareness, becomes independent terms. Then:

• T-Shirt becomes t AND shirt and also matches "Shirt rot mit T Aufdruck" (t and shirt occur, but not next to each other)
• PS5 becomes ps AND 5 and also matches "Ford Focus 125 PS Euro 5" (ps and 5 occur, but not next to each other)

We need: accept either of N alternative spellings / word groupings for one query term, and still run the field's normal linguistics (German stemming, germanNormalization) on each alternative.

Elasticsearch does this without extra query logic. Search analyzer:

word_delimiter_graph
  generate_number_parts: true
  catenate_words: true
  preserve_original: true

That filter emits a token graph: split parts as sequential tokens (t, then shirt), a catenated token that spans those positions (tshirt, positionLength = 2), optionally the original token. We query with query_string and that analyzer. Lucene QueryBuilder walks the graph. With auto_generate_synonyms_phrase_query it turns a multi-position path into a phrase, and stacked paths at the same position into alternatives. Same shape as a multi-word synonym ny, new york becoming (ny OR "new york"). Indexing ignores positionLength; the index analyzer is word_delimiter with generate_number_parts + catenate_words and no preserve_original.

On Vespa we configure the same Lucene filters on Lucene Linguistics. Query construction uses userInput (grammar: all, grammar.profile for the German search analyzer). Query-side linguistics stems and normalizes. It does not consume positionLength / alternative paths as phrases. So T-Shirt stays AND(t, shirt), not EQUIV(tshirt, phrase(t, shirt)). text() / grammar.tokenization: linguistics delegates tokenization to linguistics, but that still does not build phrases from the graph.

Describe the solution you'd like

A supported way in Lucene Linguistics / query parsing to:

1. Keep wordDelimiterGraph on the search analyzer (with preserveOriginal / catenateWords as needed).
2. Walk the resulting token graph at query time.
3. Emit phrase items for sequential split parts, and equiv / alternatives for stacked paths (catenated / original vs parts).
4. Still run the rest of the field linguistics (stemming, normalization) on each emitted term.

Minimum documents (keyword field, index-time word delimiter as above):

Expected matches:

German stemming / language profile must still run on each alternative.

Expected query tree (after linguistics, approximate Vespa items):

T-Shirt is letter + delimiter + letter, so catenateWords produces a joined alternative. PS5 / 32GB are letter/digit splits, so catenateWords does not join them. Adjacent parts are enough, because index-time splitting already stored ps + 5 next to each other for a document written PS5.

Describe alternatives you've considered

1. Custom container Searcher (what we run today). After YQL parsing it splits the raw query string on letter/digit/delimiter boundaries and replaces each split keyword item with EQUIV(catenated, phrase(parts)) or only phrase(parts). Filters, nearestNeighbor, and negations stay untouched. WordItems are created as from-query so StemmingSearcher still stems them. It works, and there is no inherent Searcher cost. The cost is that we reimplement WordDelimiter rules instead of reading Lucene's graph, and we have to special-case query-tree parents (NotItem.removeItem(0) puts a TrueItem back, so a rewrite that removes children does not terminate).

2. Application-level query rewrite before YQL. Same split/phrase logic, but duplicated per client and outside linguistics. Worse than a Searcher.

3. Port the ES analyzers via Lucene Linguistics and stop there. That covers joined-form recall for some cases. It does not give adjacency. That needs graph-to-phrase expansion, which is the auto_generate_synonyms_phrase_query piece Vespa does not have.

Additional context

Related: query-side analysis currently stems and normalizes, it does not fully tokenize via linguistics unless you use text() / grammar.tokenization: linguistics (#33540). Even with that, the missing piece is turning a WordDelimiterGraph (or any graph filter with positionLength > 1) into phrase alternatives.

Happy to share a reduced application package or the Searcher if that helps.

CC: @radu-gheorghe