[serve][llm] Governance middleware layer for Ray Serve LLM — PII detection, cost budgets, policy enforcement, and audit trails

Description

Ray Serve LLM currently exposes an OpenAI-compatible API (chat completions, completions, embeddings) with no built-in mechanism for governance, guardrails, or policy enforcement at the serving layer. Teams deploying LLMs in production — especially in regulated industries (healthcare, finance, government) — need to add PII detection, cost budgets, tool authorization, and audit logging around every inference request.

Today, the only way to achieve this is to write a custom Ray Serve deployment that wraps the engine directly (bypassing the standard LLMConfig + build_openai_app pipeline), losing all the benefits of LLMRouter, LLMServer, autoscaling, and multi-model management.

Proposed: A middleware/hook system for Ray Serve LLM that allows governance logic to execute before and after inference — similar to ASGI middleware or Ray Serve's HTTP middleware, but specifically integrated into the LLM request pipeline.

from ray import serve
from ray.serve.llm import LLMConfig, build_openai_app, LLMMiddleware

class GovernanceMiddleware(LLMMiddleware):
    """Deterministic governance — no LLM in the governance path, <2ms overhead."""
    
    async def before_inference(self, request: ChatCompletionRequest, context: RequestContext) -> ChatCompletionRequest | BlockedResponse:
        # 1. Scan input for PII (regex-based, 40+ patterns)
        pii_findings = self.scan_pii(request.messages)
        if pii_findings and self.config.mode == "ENFORCE":
            return BlockedResponse(reason="PII detected", findings=pii_findings)
        
        # 2. Check cost budget (per-user, per-session, per-model)
        if self.exceeds_budget(context.user_id, context.model):
            return BlockedResponse(reason="Budget exceeded")
        
        # 3. Evaluate access policies (which users can use which models)
        if not self.policy_allows(context.user_id, request.model):
            return BlockedResponse(reason="Model access denied by policy")
        
        return request  # Allow request to proceed
    
    async def after_inference(self, request: ChatCompletionRequest, response: ChatCompletionResponse, context: RequestContext) -> ChatCompletionResponse:
        # 1. Scan output for PII/secrets leakage
        # 2. Record token usage against cost budget
        # 3. Emit structured audit receipt (JSON with correlation ID)
        self.record_usage(context, response.usage)
        self.emit_audit_receipt(context, request, response)
        return response

# Usage: plug into standard LLMConfig pipeline
llm_config = LLMConfig(
    model_loading_config={"model_id": "gpt-serving", "model_source": "meta-llama/Llama-3-8B-Instruct"},
    accelerator_type="A10G",
)

app = build_openai_app({
    "llm_configs": [llm_config],
    "middleware": [GovernanceMiddleware(config=governance_policy)],  # <-- new
})
serve.run(app, blocking=True)

Key design principles:

• Middleware runs in the same process (no sidecar, no network hop)
• Deterministic evaluation only (regex, policy rules) — no additional LLM call
• <2ms overhead per request
• Does not interfere with vLLM/SGLang engine internals
• Composable: multiple middleware can chain (auth → governance → logging)

Use case

1. Regulated industries deploying LLMs on Ray

Healthcare companies (HIPAA), financial services (SOX, PCI-DSS), and government agencies need to prove that every LLM interaction was governed. Today they either:

• Build custom Ray Serve deployments from scratch (losing LLMRouter, autoscaling, multi-model support)
• Add a separate proxy/gateway in front of Ray Serve (adding latency, operational complexity, and a failure point)

A middleware layer inside Ray Serve LLM would let them use the standard LLMConfig pipeline with governance built in.

2. Multi-tenant LLM platforms

Companies serving multiple customers/teams from shared GPU infrastructure need:

• Per-tenant cost budgets with hard enforcement (stop serving when budget is exhausted)
• Per-tenant model access policies (tenant A gets Llama-70B, tenant B gets Llama-8B only)
• PII isolation (scan inputs/outputs to prevent data leakage across tenants)

The LLMRouter already handles routing, but there's no governance layer for access control or cost caps.

3. MCP tool governance in agentic deployments

As Ray Serve LLM adds tool-calling and agentic capabilities, there's a need to control which tools agents can invoke, validate tool arguments before execution, and scan tool outputs before they re-enter the context window. A middleware layer is the natural extension point for this.

4. Compliance evidence for SOC2/ISO 27001 audits

Enterprises need structured audit records proving that:

• Every request was scanned for sensitive data
• Access policies were evaluated deterministically
• Cost budgets were enforced
• Blocked requests have documented reasons

Current Ray Serve metrics (Prometheus/Grafana) cover performance but not governance decisions.

Existing community demand:

The Ray Discuss thread on preprocessing highlights that users already want more control over the request pipeline. A governance middleware would address this need specifically for security/compliance use cases.

I've built TealTiger (Apache 2.0), a deterministic AI governance SDK already integrated with 15+ agent frameworks (LangChain, CrewAI, AG2, Haystack, n8n). Happy to contribute a Ray Serve LLM middleware integration that wraps TealTiger's governance engine.

Key properties:

• No LLM in the governance path — all evaluation is deterministic
• <2ms latency overhead
• 40+ PII detection patterns with confidence scoring
• ENFORCE / MONITOR / REPORT_ONLY modes
• Structured JSON decision receipts with OpenTelemetry-compatible trace IDs
• Covers 7/10 OWASP AI Security Issues

Labels

enhancement, triage, serve