Metric integrity: faithfulness and context-recall divide by emitted verdicts not submitted inputs; errors become NaN and exit headline; context embedded unsanitized into judge prompts; extract_json takes first structure

Summary

Five defects in ragas metric scoring at pin 298b682, all with deterministic rule-based judge stubs.

1. Faithfulness divides by emitted verdicts, not submitted statements

The denominator is the number of verdicts the judge produced, never reconciled with the input. A judge that emits one verdict for a five-statement answer scores 1.0; honest scoring gives 0.2.

2. Context recall divides by emitted classifications

Same family: the denominator shrinks to the classifications the judge produced, silently inflating recall.

3. Per-sample errors become NaN and exit the headline silently

Default raise_exceptions=False converts errors to NaN; the headline uses np.nanmean. A 90 percent junk dataset prints a perfect score.

4. Retrieved context embedded into judge prompts without sanitization

A SYSTEM OVERRIDE directive in retrieved context ships verbatim into the judge prompt.

5. extract_json takes the first JSON structure

Preliminary verdicts (0,0) followed by final (1,1) score 0.0; reversed order scores 1.0. The final block is never consumed.

All five reproduce with offline rule-based stubs. Note #2983 (cross-model cache) is a different surface from these metric-level issues.