RemoveSearchAttributes skips the secondary store under dual visibility
With dual visibility on two Elasticsearch stores, removing a custom search attribute removes it from the primary's metadata twice and never touches the secondary's. The call reports success, and the attribute reappears when the secondary is promoted.
Reproduction (v1.31.0, two ES clusters)
temporal operator search-attribute create --name DualVisRepro --type Keyword→ cluster metadata has the attribute under both index names; mapping present on both clusters. Correct.
temporal operator search-attribute remove --name DualVisRepro --yes→ Search attributes have been removed, exit 0. search-attribute list no longer shows it. → cluster metadata still has it under the secondary's index name.
Then swap visibilityStore / secondaryVisibilityStore in static config and restart the frontends (i.e. promote the secondary):
→ DualVisRepro is back in search-attribute list.
To inspect directly: SELECT encode(data,'escape') FROM cluster_metadata_info; — each index has its own attribute map in that blob.
Fix
// operator_handler.go:385 (and :421)
- indexName := h.visibilityMgr.GetIndexName()
+ indexName := visManager.GetIndexName()ListSearchAttributes (:484) is worth a look too — it silently describes only the primary, which is what makes this undetectable.
Current workaround
There's no API-level option — RemoveSearchAttributesRequest carries only search_attributes and namespace, so nothing you pass can target the secondary.
If you're mid-migration, just re-run the removal after promotion. Once the old secondary becomes the primary, the same temporal operator search-attribute remove applies to it.
Source: temporalio/temporal