Contains filter on description matches nothing on five search indexes (index_options strips positions)
A Contains filter on description returns nothing on five search indexes. There is no error and
no warning — the query is valid, it just cannot match.
Affected (top-level description on a dataAsset index):
| Index | index_options |
|---|---|
en/glossary_term_search_index |
docs |
ru/glossary_term_search_index |
docs |
jp/metric_search_index |
docs |
zh/api_endpoint_search_index |
freqs |
zh/metric_search_index |
docs |
Root cause
A Contains filter compiles to a phrase query, and a phrase query needs token positions in the
inverted index. Elasticsearch's default index_options for a text field is positions; these five
mappings explicitly downgrade it:
docs— stores neither term frequencies nor positions;freqs— stores frequencies but no positions.
Either way the phrase query has nothing to match against, so it silently returns zero hits.
Three of the five compound the contradiction by also declaring "term_vector": "with_positions_offsets" — asking for positions in the term vector while index_options strips them
from the index.
The analysis and mapping blocks are authored by copying an existing entity's file, which is how a setting like this spreads to some entities and not others.
How we know it is fixed
A property test rather than a hard-coded list:
IndexAnalyzerMappingTest#dataAssetDescriptionsSupportPhraseQueries walks every index carrying the
dataAsset parent alias, across en/ru/jp/zh, and fails naming any whose top-level
description.index_options is docs or freqs. On main it reports exactly the five above.
Written as a property so a future entity added by copy-paste is caught by the same test, rather than needing this issue filed again.
Reindex required: IndexMappingVersionTracker hashes each mapping file, so the affected indexes
are flagged automatically — but an existing deployment needs that reindex before descriptions become
searchable by phrase.
Source: open-metadata/OpenMetadata