[istio 1.30.0] legacy `istio.metadata_exchange` and native `io.istio.peer_metadata` both write `FilterState` key `peer_not_found` with incompatible types, tripping `envoy_bug` on TCP egress to `MESH_EXTERNAL` destinations

Is this the right place to submit this?

• This is not a security vulnerability or a crashing bug
• This is not a question about how to use Istio

Bug Description

After upgrading istiod from 1.29.2 to 1.30.0, sidecars and gateways consistently log the following envoy_bug whenever they open a TCP connection to a MESH_EXTERNAL destination (e.g. an external ServiceEntry such as www.gstatic.com):

error envoy envoy_bug external/envoy/source/common/stream_info/filter_state_impl.cc:75
  envoy bug failure: false. Details: FilterStateAccessViolation:
  FilterState::setData<T> called twice with different state types: peer_not_found.

stacktrace for envoy bug:
 #0 Envoy::Extensions::NetworkFilters::PeerMetadata::UpstreamFilter::populateNoPeerMetadata()
 #1 Envoy::Extensions::NetworkFilters::PeerMetadata::UpstreamFilter::consumePeerMetadata()
 #2 Envoy::Extensions::NetworkFilters::PeerMetadata::UpstreamFilter::onData()
 #3 Envoy::Network::FilterManagerImpl::onContinueReading()
 #4 Envoy::Network::ConnectionImpl::onReadReady()
 ...

Root cause (control-plane side)

Two filters end up running on the same upstream connection and both call FilterState::setData<T> with the key peer_not_found, but with different C++ object types:

1. Legacy TCP MX filter istio.metadata_exchange (enable_discovery: true, protocol: istio-peer-exchange), unconditionally added to the cluster's filters[] list by applyMetadataExchange in pilot/pkg/networking/core/cluster_builder.go:430:

   go复制

   func (cb *ClusterBuilder) applyMetadataExchange(c *cluster.Cluster) {
       if features.MetadataExchange {
           c.Filters = append(c.Filters, xdsfilters.TCPClusterMx)
       }
   }

   For a MESH_EXTERNAL upstream the peer never speaks the istio-peer-exchange wire format, so discovery always fails and the filter stamps peer_not_found with the legacy CEL/WASM-style C++ state object.

2. Native upstream peer-metadata filter io.istio.peer_metadata (proxy-side), which runs because Istio 1.30 wires InternalUpstreamTransport + passthroughMetadata: istio.peer_metadata into the cluster (pilot/pkg/networking/util/internal_upstream.go). Its populateNoPeerMetadata() path then writes peer_not_found with the new native typed object.

The new storage was advertised in releasenotes/notes/telemetry-cel.yaml:

Peer metadata is now stored in filter_state.downstream_peer and filter_state.upstream_peer instead of filter_state["wasm.downstream_peer"] and filter_state["wasm.upstream_peer"].

FilterState::setData<T> enforces "one C++ type per key for the life of the connection," so the second writer triggers the ENVOY_BUG. The condition fires on every TCP connection to a non-MX-speaking upstream — i.e. essentially every MESH_EXTERNAL TCP destination in production meshes.

What is not the cause

• Diff of pilot/ between tags 1.29.2 and 1.30.0 shows the legacy TCPClusterMx definition (pilot/pkg/xds/filters/filters.go) and internal_upstream.go are byte-identical. The behavioral change came in via the proxy SHA bump in istio.deps:
  复制

  - "lastStableSHA": "af30be60b7c35f2aceaea1b7382c7fbf12aa5e67"   # proxy 1.29.2
  + "lastStableSHA": "7319e99e3b3a4cd0f44208e18af02c7399eca1b0"   # proxy 1.30.0

  So the type collision is an interaction between the still-emitted legacy MX cluster filter and the new native peer-metadata storage type in istio/[email protected].

Precedent that this was a known surface

The HTTP-header equivalent of this problem was already fixed for 1.30 in releasenotes/notes/skip-mx-headers.yaml (MetadataExchangeHeaders=IN_MESH so x-envoy-peer-metadata* headers aren't sent to mesh-external destinations). The TCP cluster-filter equivalent appears to have been overlooked.

cluster_builder.go:395-408 even already annotates the cluster metadata with istio.external = true (for MeshExternal) and istio.disable_mx = true (for allInstancesHBONE) — the intent to suppress MX on these clusters is already encoded in the control plane. Only the legacy filter is no longer being suppressed in the proxy on the external: true path.

Reproduction

1. Install Istio 1.30.0 (default config; no special envs).
2. Apply a MESH_EXTERNAL ServiceEntry for any external host, e.g.:
   yaml复制

   apiVersion: networking.istio.io/v1beta1
   kind: ServiceEntry
   metadata:
     name: gstatic
   spec:
     hosts: ["www.gstatic.com"]
     ports:
     - number: 10000
       name: tcp
       protocol: TCP
     location: MESH_EXTERNAL
     resolution: DNS

3. From any injected workload, open a TCP connection to www.gstatic.com:10000.
4. Observe the envoy_bug log shown above and the corresponding connection failure. server.envoy_bug_failures counter increments.

Expected behavior

Either:

• The legacy istio.metadata_exchange cluster filter should not be attached to MESH_EXTERNAL clusters (mirror the skip-mx-headers fix for the TCP path), or
• The 1.30 proxy should honor the existing cluster.metadata.istio.external == true hint (same way it honors disable_mx) and skip the upstream peer-metadata filter's setData for peer_not_found.

Then thread service.MeshExternal (and opts.meshExternal for the subset path) through the three call sites at lines 339, 392, and 771.

Workaround

Setting PILOT_ENABLE_METADATA_EXCHANGE=false on istiod removes the legacy cluster filter and eliminates the envoy_bug. Verified by re-dumping the affected cluster with istioctl pc c <pod> -o yaml --fqdn 'outbound|10000||www.gstatic.com' — the filters: [istio.metadata_exchange ...] block is gone, and server.envoy_bug_failures stays flat. Trade-off: loss of MX-derived workload-attribution labels on plain-TCP and mTLS-TCP in-mesh telemetry (HTTP traffic and HBONE-tunneled traffic are unaffected; identity from mTLS SPIFFE SAN is unaffected). PEER_METADATA_DISCOVERY=true (MDS) is the documented path to recover those labels.

Version

$ istioctl version
client version: 1.30.0
control plane version: 1.30.0
data plane version: 1.30.0 (2964 proxies)

$ kubectl version
Client Version: v1.34.2
Kustomize Version: v5.7.1
Server Version: v1.34.2

Additional Information

Affected cluster dump (excerpt)

istioctl pc c <pod>.<ns> -o yaml --fqdn 'outbound|10000||www.gstatic.com':

- altStatName: outbound|10000||www.gstatic.com;
  filters:
  - name: istio.metadata_exchange
    typedConfig:
      '@type': type.googleapis.com/udpa.type.v1.TypedStruct
      typeUrl: type.googleapis.com/envoy.tcp.metadataexchange.config.MetadataExchange
      value:
        enable_discovery: true
        protocol: istio-peer-exchange
  metadata:
    filterMetadata:
      istio:
        external: true    # <-- control plane already marks this; proxy ignores
        services:
        - host: www.gstatic.com
          name: www.gstatic.com
          namespace: <ns>
  transportSocketMatches:
  - match: { tunnel: http }
    name: hbone
    transportSocket:
      name: envoy.transport_sockets.internal_upstream
      typedConfig:
        '@type': type.googleapis.com/envoy.extensions.transport_sockets.internal_upstream.v3.InternalUpstreamTransport
        passthroughMetadata:
        - { kind: { cluster: {} }, name: istio.peer_metadata }
        - { kind: { host:    {} }, name: istio.peer_metadata }
        transportSocket:
          name: envoy.transport_sockets.raw_buffer
          typedConfig:
            '@type': type.googleapis.com/envoy.extensions.transport_sockets.raw_buffer.v3.RawBuffer
  - match: {}
    name: tlsMode-disabled
    transportSocket:
      name: envoy.transport_sockets.raw_buffer
      typedConfig:
        '@type': type.googleapis.com/envoy.extensions.transport_sockets.raw_buffer.v3.RawBuffer
  type: STRICT_DNS

After PILOT_ENABLE_METADATA_EXCHANGE=false, the filters: block above is gone and the envoy_bug stops firing; the transportSocketMatches block is unchanged.

Pointers into the source

• Legacy filter definition (unchanged between 1.29.2 and 1.30.0): pilot/pkg/xds/filters/filters.go (TCPClusterMx)
• Injection site: pilot/pkg/networking/core/cluster_builder.go:430 (applyMetadataExchange)
• Three call sites that should pass meshExternal: cluster_builder.go:339, :392, :771
• InternalUpstreamTransport setup (unchanged): pilot/pkg/networking/util/internal_upstream.go
• External/disable_mx metadata annotation (already present): cluster_builder.go:395-408
• Proxy SHA bump driving the regression: istio.deps PROXY_REPO_SHA af30be60..7319e99e
• Existing partial fix for the HTTP path: releasenotes/notes/skip-mx-headers.yaml
• Release note describing the new storage location: releasenotes/notes/telemetry-cel.yaml

istioctl bug-report

Reporter: please attach istioctl bug-report output. I'm filing this on behalf of the operator and don't have a fresh archive available at write time. Happy to add one in a follow-up comment.