New metric: PII Leakage detection for RAG/agentic responses

Describe the Feature Ragas doesn't currently have a metric that checks whether a generated response discloses personally identifiable information (PII) — names, addresses, medical/financial details, government IDs, contact info. I'd like to propose a PIILeakage metric that scores a response for this, ranging 0 (heavy disclosure) to 1 (none detected).

Why is the feature important for you? RAG and agentic systems frequently answer questions over documents that themselves contain sensitive data — patient records, HR files, financial statements, support tickets. Retrieval is supposed to surface relevant content, but nothing currently checks whether the generation step over-shares — e.g., a user asks "does this patient have an active policy?" and the model answers with the patient's full diagnosis instead of yes/no.

Concrete use cases:

• Healthcare RAG: verifying a clinical Q&A assistant doesn't restate a patient's diagnosis/medications when the question didn't ask for them.
• HR/internal-docs chatbots: catching accidental disclosure of salary, SSN, or performance-review content pulled from retrieved context.
• Financial services support bots: flagging account numbers or balances surfacing in a response to an unrelated question.
• General regression testing: teams building RAG eval suites could use this alongside Faithfulness/ContextPrecision to catch privacy regressions the same way those catch factual regressions.

Ragas has strong correctness/faithfulness coverage but no first-party safety/privacy metric, even though safety evaluation is a common requirement for regulated-domain RAG deployments.

Additional context Open to discussing scope — e.g. whether this should catch only clearly-identifiable PII (SSNs, emails, phone numbers) or also more ambiguous cases like several individually-harmless details that together identify someone. A related but distinct follow-up (credential/secret leakage — API keys, passwords, tokens) is a separate concern from PII and would make more sense as its own metric later, not folded into this one. Happy to submit a PR with an implementation once there's agreement on the general direction and rough scope.