Search indexing fails with 'Forbidden access' on cluster-alias-scoped search roles

Description

On any deployment with a non-empty clusterAlias and a search role scoped to that prefix — every shared-tenancy Collate Cloud cluster — the search indexing job dies before it reads a single record:

RuntimeException: Cannot determine the live chunk target for <clusterAlias>_data_asset_embeddings_chunks
Caused by: java.io.IOException: Forbidden access

The fleet report shows the signature clearly: failed · 0 failed records.

Root cause

OpenSearchVectorService.resolveLiveChunkTargetStrict probes the chunk read alias with client.indices().existsAlias(a -> a.name(base)). With only name set, ExistsAliasRequest builds HEAD /_alias/{name} — no index in the path. OpenSearch resolves a no-index alias-get against _all, so it requires indices:admin/aliases/get cluster-wide. A role granting indices_all on <clusterAlias>* only is denied with a 403.

That 403 can never degrade to false: RestClientTransport.getHighLevelResponse raises TransportException("Forbidden access") (an IOException) on status 403 before the BooleanEndpoint status→boolean mapping runs.

requireResolvedLiveChunkTarget wraps it and throws, and RecreateWithEmbeddings.reCreateIndexes calls recreateChunkIndexIfFullRun before super.reCreateIndexes(entities) — so the exception aborts the entire reindex at startup.

Introduced by #30364, which both un-gated the staged chunk recreate (previously dead code behind a flag that could never be true) and replaced a swallowed GET /_alias/{base} with this strict probe. Before that commit the same 403 was caught at debug level and resolution fell through to the index-scoped HEAD /{base}, which the role does allow.

Same class of problem elsewhere

• OpenSearchIndexManager.getIndicesByAlias / ElasticSearchIndexManager.getIndicesByAlias — both the existsAlias pre-probe and GetAliasRequest.of(g -> g.name(...)) are the cluster-wide form. The 403 is swallowed, so it silently returns an empty index set instead of failing.
• SearchRepository.indexTemplatesMatch reads om_*, which returns every co-tenant's index templates on a shared cluster.

Expected

Every request names a concrete index or pattern beginning with the cluster alias; nothing resolves to _all. No cluster-wide grant should be required beyond what index-template management already needs.

Affected

OpenSearch or Elasticsearch, naturalLanguageSearch.semanticSearchEnabled: true, non-empty clusterAlias, and a search role confined to that prefix. Deployments whose search user is unrestricted are unaffected, which is why this only shows up on shared-tenancy clusters.