Eval summary pass counts do not match row-level results

Author: pamelafoxCreated Jun 12, 2026Updated Jun 17, 2026

Problem\nIn committed eval artifacts, summary.json can disagree with eval_results.jsonl for gpt_relevance pass counts. For example, rows with gpt_relevance: 3.0 and relevance_threshold: 3 are marked as relevance_result: pass, but the summary reports fewer passes (e.g. 47/50 instead of 50/50).\n\n## Expected behavior\nsummary.json and eval_results.jsonl should use the same pass/fail rule and report consistent counts.\n\n## Notes\nThis appears to be an evaluator/pipeline issue rather than a problem with the PR that refreshed historical eval artifacts.

Source: Azure-Samples/azure-search-openai-demo