One WasmPlugin CR with a missing wasm file breaks ALL routes on the gateway (500 / NFCF), not just its own
Summary
A single WasmPlugin CR whose url points to a file that does not exist on the gateway causes all requests through the gateway to fail with 500 (response_flags: NFCF) — including requests on routes that have no wasm plugin attached at all. One broken plugin takes down all traffic on the listener instead of only affecting its own matched routes.
Environment
- Higress controller/pilot/gateway: official
v2.2.4images (higress-registry.cn-hangzhou.cr.aliyuncs.com/higress/{higress:2.2.4,pilot:3498bc95…,gateway:3498bc95…}) - Plugin wasm files served through the mounted volume (
global.volumeWasmPlugins=true, the standardmake higress-wasmplugin-testkind setup) - k8s v1.34 (kind)
Steps to reproduce
- Install higress with
global.volumeWasmPlugins=trueand an ai-proxy wasm present at/opt/plugins/wasm-go/extensions/ai-proxy/plugin.wasm. Verify a route through ai-proxy works (200). - Create a
WasmPluginCR referencing a file that does not exist (e.g. left over from a test run whose plugin was never built):
apiVersion: extensions.higress.io/v1alpha1
kind: WasmPlugin
metadata:
name: cache-control
namespace: higress-system
spec:
defaultConfig: {}
url: file:///opt/plugins/wasm-go/extensions/cache-control/plugin.wasm- Send any request through the gateway — including to an ingress with no WasmPlugin matching it.
Observed behavior
Every request through the listener now returns an immediate empty-body 500:
{"response_code":"500","response_flags":"NFCF","route_name":"<matched route>",
"upstream_cluster":"outbound|…","upstream_host":"-","response_code_details":"","bytes_sent":"0"}The upstream cluster is healthy and present in the cluster manager (/clusters shows it with a healthy endpoint); the route resolves correctly. Envoy debug logs show the local reply is prepared right after the envoy.filters.http.custom_response filter records need_fallback: true dynamic metadata:
[warning][config] delta config for type.googleapis.com/envoy.config.core.v3.TypedExtensionConfig rejected: Invalid path: /opt/plugins/wasm-go/extensions/cache-control/plugin.wasmdynamicMetadata: filter_metadata { key: "envoy.filters.http.custom_response" value { fields { key: "need_fallback" value { bool_value: true } } } }
[debug][http] Preparing local reply with detailsDeleting the broken WasmPlugin CR immediately restores all traffic (verified: same requests go from 500/NFCF back to 200).
Expected behavior
A wasm plugin whose code cannot be loaded should fail closed for its own matched routes (or reject the config at admission), but must not degrade unrelated routes that share the listener.
Impact
Besides the production-robustness concern, this makes e2e/conformance workflows fragile in a non-obvious way: an interrupted make higress-wasmplugin-test run leaves stale WasmPlugin CRs behind (for plugins whose wasm files are only built on demand); every subsequent test run then fails all assertions with 500/NFCF regardless of the code under test, and the root cause is invisible unless one looks for Invalid path warnings in the gateway log.
Source: higress-group/higress