OTel: successful plugin-terminated requests (`StatusRespond`) are recorded as span status Error ("response succeed")

Summary

When a custom Go plugin (custom_middleware, driver goplugin) writes a successful (2xx) response, the root OpenTelemetry server span is marked with status Error and description response succeed, even though the request succeeded with an HTTP 200. The span still carries http.status_code: 200.

The root cause is an ordering issue in TraceMiddleware.ProcessRequest: it sets the span status to Error from the raw middleware error before the generic middleware runner (createMiddleware) neutralizes the internal ErrResponseSucceed sentinel. The sentinel is Tyk's success signal, not a failure, and createMiddleware already treats it as non-error — but the trace wrapper stamps the span first.

Net effect: any API served primarily by a Go plugin that writes the response shows a large, misleading error rate on OpenTelemetry backends (Datadog APM, Jaeger, Grafana Tempo, etc.), because the backend derives the error from the span status, not from http.status_code.

Affected version

• Tyk Gateway v5.15.0 (confirmed by reading source at the v5.15.0 tag).
• Also reproduces on v5.13.x; not fixed by the 5.14 / 5.15 releases. The relevant code is unchanged across these versions.
• OpenTelemetry enabled (opentelemetry.enabled: true), any exporter.
• otelhttp instrumentation go.opentelemetry.io/contrib/instrumentation/net/http/otelhttp (observed v0.67.0).

How to reproduce

1. Run an OSS gateway with OpenTelemetry tracing enabled and an OTLP exporter (or an OTel collector) attached.
2. Deploy a Tyk Classic API whose response is produced by a custom Go plugin attached as custom_middleware.post (driver goplugin) — i.e. the plugin calls rw.Write(...) / rw.WriteHeader(200) and returns, short-circuiting the reverse proxy. (An AWS Lambda native-invoke plugin is one common example, but any response-writing Go plugin triggers it.)
3. Send a request that the plugin serves successfully (HTTP 200).
4. Inspect the resulting http.server.request span.

Observed span

http.status_code:        200
otel.status_code:        Error
otel.status_description: response succeed

(The request genuinely succeeded — 200 with a real response body.)

Expected span

http.status_code:        200
otel.status_code:        Unset (or Ok)

Per the OTel HTTP semantic conventions, a 2xx server response should not set span status to Error.

Root cause

1. A plugin-written 2xx returns the ErrResponseSucceed sentinel

gateway/mw_go_plugin.go — handlePluginResponse returns the sentinel with middleware.StatusRespond on the success path:

// record 2XX to analytics
successHandler.RecordHit(r, analytics.Latency{ ... }, rw.statusCodeSent, rw.getHttpResponse(r), false)

// no need to continue passing this request down to reverse proxy
return ErrResponseSucceed, middleware.StatusRespond

ErrResponseSucceed is explicitly a benign domain-typed signal, not a failure:

var (
    ErrResponseSucceed = errpack.New(
        "response succeed",
        errpack.WithType(errpack.TypeDomain),
        errpack.WithLogLevel(logrus.TraceLevel),
    )
    ErrResponseErrorSent = errpack.New(
        "plugin error",
        errpack.WithType(errpack.TypeDomain),
        errpack.WithLogLevel(logrus.DebugLevel),
    )
)

2. TraceMiddleware stamps the span Error before the sentinel is swallowed

gateway/middleware.go — TraceMiddleware.ProcessRequest (OTel branch) sets the span status from the raw error, with no exclusion for ErrResponseSucceed:

span := otel.SpanFromContext(r.Context())
err, i := tr.TykMiddleware.ProcessRequest(w, r, conf)
if err != nil && span != nil {
    span.SetStatus(otel.SPAN_STATUS_ERROR, err.Error())   // <-- err == ErrResponseSucceed here
}

3. …but createMiddleware already knows the sentinel is not an error

A few frames up, the generic runner neutralizes exactly this sentinel:

err, errCode := mw.ProcessRequest(w, r, mwConf)

// Workaround
// ProcessRequest signature is too narrow ...
if errors.Is(err, ErrResponseSucceed) {
    err = nil
}

So the sentinel is correctly treated as non-error for request handling, but TraceMiddleware (which wraps the plugin middleware and runs inside ProcessRequest) has already written Error to the span by the time this runs. The trace wrapper simply didn't get the same exclusion the runner has.

The response-side path has the same shape (handleOtelTracedResponse also does if err != nil { span.SetStatus(otel.SPAN_STATUS_ERROR, err.Error()) }), so any fix should consider both.

Suggested fix

Exclude the benign domain sentinels in TraceMiddleware.ProcessRequest (and, symmetrically, handleOtelTracedResponse), mirroring what createMiddleware already does:

err, i := tr.TykMiddleware.ProcessRequest(w, r, conf)
if err != nil && span != nil &&
    !errors.Is(err, ErrResponseSucceed) &&
    !errors.Is(err, ErrResponseErrorSent) {
    span.SetStatus(otel.SPAN_STATUS_ERROR, err.Error())
}

A more general variant would skip span-error status for any errpack.TypeDomain error (via errpack's TypeOf), since those are classified as domain-level signals rather than failures. Either approach fixes the reported behavior; naming the two sentinels explicitly is the minimal, lowest-risk change.

Note: ErrResponseErrorSent accompanies a real 4xx/5xx the plugin wrote itself, so if you prefer to keep the span red for genuine plugin-sent errors, exclude only ErrResponseSucceed. The distinguishing signal is the HTTP status code (rw.statusCodeSent), which is already 2xx on the ErrResponseSucceed path and

= 400 on the ErrResponseErrorSent path.

Impact / workaround

• Impact: OTel backends over-report the error rate for any plugin-served API. On a service whose traffic is mostly plugin-written 200s, the APM service page can show a majority "error" rate while every request is a successful 200.
• Workaround (consumer side): derive the true error signal from http.status_code (e.g. http.status_code:[500 TO 599]) rather than the span error flag when building monitors/dashboards. This does not remove the false error status on the spans themselves.

Environment

• Tyk Gateway: v5.15.0 (OSS), also seen on v5.13.x
• Deploy: file-based (use_db_app_configs: false), Classic API definitions
• Plugin: native Go plugin (goplugin) attached as custom_middleware.post, writing the response and short-circuiting the reverse proxy
• OTel: opentelemetry.enabled: true, OTLP/gRPC exporter
• Runtime: go1.26.7 linux/arm64