Bug Report: ElasticSearch indexer deletes stale indices immediately after alias rotation, failing in-flight search queries with 500

Issue Labels

• Please familiarize yourself with the issue labels used in this project: LABELS.md

Search Terms

index_not_found_exception, search_phase_execution_exception, MissingIndexError, Removing stale search indices, alias rotation, stale index deletion, search 500 during indexing

️ Project Area

Search

External Integration

AWS

Description & Context

ElasticSearchSearchEngineIndexer.finalize() rotates the search alias onto the newly built index and then immediately deletes the stale indices. A search query that started before the rotation is already bound to the old concrete index, so when that index is deleted mid-flight the query fails with index_not_found_exception and the user receives a 500.

There is no grace period between the alias swap and the deletion, so any query in flight during that window fails.

Evidence

Observed continuously in production for at least 4.5 months (as far back as our log retention goes), at a rate of 15–50 occurrences per week — roughly 1 in 380 searches.

A single incident, showing the deletion directly causing the failures:

11:10:21.979  info   Indexing completed for index techdocs-index__1788347358606 of techdocs in 63.4s
                     (documentType: techdocs, plugin: search)

11:20:19.334  info   Removing stale search indices
                     (documentType: techdocs, plugin: search)

11:20:21.153  error  Failed to query documents for indices techdocs__search
                     search_phase_execution_exception: [index_not_found_exception]
11:20:21.266  error  Failed to query documents for indices techdocs__search
11:20:21.445  error  Failed to query documents for indices techdocs__search
11:20:21.671  error  Failed to query documents for indices techdocs__search
11:20:22.147  error  Failed to query documents for indices techdocs__search
11:20:22.382  error  Failed to query documents for indices techdocs__search
11:20:22.622  error  Failed to query documents for indices techdocs__search

11:20:27.073  error  Request failed with status 500  Missing index for techdocs__search
11:20:28.704  error  Request failed with status 500
11:20:29.955  error  Request failed with status 500

The index was created at 11:10:21, lived exactly one 10-minute collator cycle, and was deleted at 11:20:19. Query failures begin 1.8 seconds later, and 500s reach users immediately after.

The underlying error from OpenSearch (endpoint redacted):

{
  "error": {
    "type": "search_phase_execution_exception",
    "phase": "fetch",
    "caused_by": {
      "type": "index_not_found_exception",
      "reason": "no such index [pBSEXqABRDbtk34rDuds]"
    }
  },
  "status": 404
}

Note "phase": "fetch" — the query had already matched documents and failed on the way back to read them. ignore_unavailable=true&allow_no_indices=true are already sent on every query by ElasticSearchClientWrapper.search(), but they only apply while resolving index names at the start of the request, so they do not cover this.

When this was introduced

Stale index deletion was added deliberately in @backstage/[email protected] (Backstage v1.28.0, b186701), marked BREAKING, with the rationale:

An indexer using the some-type-index__* pattern will remove indices matching this pattern after indexation to prevent stale indices leading to shards exhaustion.

That change is sound — the gap is only that deletion happens with no grace period for queries already in flight.

Still present in 2.0.0 and on master (verified against source, not just the changelog).

Two secondary issues that made this hard to diagnose

1. ElasticSearchSearchEngine.query() only recognises a top-level index_not_found_exception:

   typescript复制

   if (error.meta?.body?.error?.type === 'index_not_found_exception') {
     throw new MissingIndexError(...);
   }

   When the same condition arrives nested inside search_phase_execution_exception (as above), it is not recognised. The existing unit test only covers the top-level shape.

2. The fallback discards the error entirely:

   typescript复制

   return Promise.reject({ results: [] });

   The rejected value is a plain object with no message, so the search router logs There was a problem performing the search query: undefined. This is why the failure is effectively undiagnosable from logs alone.

Possibly related, previously closed without diagnosis

• #25797 — same symptom, closed stale
• #25339 — same symptom, closed stale

Expected Behavior

A search query that is in flight when a collator finishes indexing should complete successfully — either against the index it originally resolved, or against the newly rotated one. Rotating the alias should not cause concurrent queries to fail with a 500.

Reproduction Repo

No response

Reproduction steps

This is a timing window rather than a deterministic failure, so reproduction requires concurrent load during an alias rotation. Observed in production rather than in an isolated repro.

1. Run Backstage with @backstage/plugin-search-backend-module-elasticsearch against Elasticsearch or OpenSearch.

2. Configure a collator with a short schedule and enough documents that indexing takes several seconds, e.g.:

   yaml复制

   search:
     collators:
       catalog:
         schedule:
           frequency: { minutes: 1 }
           timeout: { minutes: 5 }

3. Run a continuous loop of search queries against /api/search/query (queries that take longer widen the window — the default multi_match over fields: ["*"] with fuzziness: auto plus highlighting takes several seconds on a large index).

4. Watch the backend logs for Removing stale search indices.

5. Within a few seconds of each of those lines, queries fail with search_phase_execution_exception / index_not_found_exception and the API returns 500.

Have you read the Code of Conduct?

• I have read the Code of Conduct

Are you willing to submit PR?

Yes, and I have enough information to get started