Request for maintainer review: pinned Portkey Gateway OSS 1.15.2 conformance measurements

I maintain a small OpenAI-compatible HTTP conformance benchmark and measured two pinned Portkey Gateway OSS configurations. I’m asking you to check my configuration, interpretation, and raw artifact.

I wrote the benchmark and maintain LLM-Shield-Proxy, one of the gateways it measures, so I have an obvious conflict of interest. Each row below is one maintainer-produced run and is unreplicated. I am asking for corrections or a reproduction, not endorsement.

What I ran

how the gateway was directed to the capture: x-portkey-provider: openai x-portkey-custom-host: http://127.0.0.1:8765/v1 For the default run: No x-portkey-config header and no hooks. For the redaction run

{
  "before_request_hooks": [
    {
      "type": "guardrail",
      "id": "conformance-pii-redact",
      "deny": false,
      "checks": [
        {
          "id": "default.regexReplace",
          "parameters": {
            "rule": "/[A-Za-z0-9._%+-]+@[A-Za-z0-9.-]+\\.[A-Za-z]{2,}|\\b(?:\\d{4}[- ]?){3}\\d{4}\\b|\\b\\d{3}-\\d{2}-\\d{4}\\b/g",
            "redactText": "[REDACTED]"
          }
        }
      ]
    }
  ]
}

Is this a correct use of default.regexReplace in Portkey Gateway OSS 1.15.2, and is it fair to interpret the resulting replacement as one-way?

• Self-hosted Portkey Gateway OSS 1.15.2 from the official image portkeyai/gateway@sha256:97f094d9c8a764cbfaa2a7138c0017b247ca923bb06db1b4c13b7f8a33b5200d
• The official image’s shipped dependency tree, under Node 20.19.6
• Local loopback capture, three iterations
• Default configuration with no guardrail enabled
• A redaction configuration using one default.regexReplace hook with tester-authored patterns

This was not Portkey’s hosted service and used no Portkey account.

The tester-authored regex run measures the OSS guardrail transformation engine, not Portkey’s own PII detector.

Three iterations are a conformance smoke test, not a performance benchmark. I am making no speed or latency claim. I am also not presenting the fresh npm dependency-tree streaming problem as a Portkey defect; the measured target used the official image’s shipped dependency tree.

Result summary

The default configuration had no guardrail enabled. Its outcome is redaction-not-enabled: a statement about that configuration, not a product verdict.

With default.regexReplace, leaked_entity_types was empty: no protected fixture value reached the configured capture upstream. The outcome is no-leak-profile-not-met because replacement was one-way and the original values were not restored to the client. This is not a privacy failure.

All links below are pinned to merged PR #22’s head commit, 1e1f72846e071367c23521d2b7bffbde639c39d4:

• Configuration record
• Default raw artifact — SHA-256 acd79ad14b76ff6f9c2f6aa1d8659abf351394cb88beb9edae46a3e8ef8196e8
• regexReplace raw artifact — SHA-256 b72a949c8fcf3a5f5b858f8c90c16a668c1848bacac8668d48e84bf8c809f19f
• HTTP-profile methodology
• Fixture threat model
• Report schema
• Reproduction instructions

Could you please check three things?

1. Is the pinned official-image setup a fair representation of Portkey Gateway OSS 1.15.2?
2. Is my interpretation of default.regexReplace as a one-way transformation—and not a measurement of Portkey’s PII detector—fair?
3. Given the raw artifacts, are redaction-not-enabled and no-leak-profile-not-met fair outcome classifications?

If anything is wrong, please tell me what to correct or provide a reproduction that reaches a different result. I will correct the public record.