Replace RAGAS synthetic ground-truth generation with Microsoft Foundry data generation service

This issue is for a:

• feature request

Summary

Replace the RAGAS-based synthetic ground-truth generator in evals/generate_ground_truth.py with the Microsoft Foundry data generation service (SimpleQnA job with FileDataGenerationJobSource), per Generate a synthetic evaluation dataset (preview).

Motivation

• RAGAS pulls in a heavy dependency chain (ragas, langchain, rapidfuzz) and persists a 28MB knowledge-graph artifact (evals/ground_truth_kg.json).
• The Foundry service is the first-party, supported path for bootstrapping evaluation datasets from reference material, and keeps the eval stack aligned with the rest of the Azure AI Foundry tooling this sample uses.

Current state (what RAGAS does today)

• evals/generate_ground_truth.py: fetches chunks from the Azure AI Search index, builds a RAGAS KnowledgeGraph, applies default_transforms (LLM + embeddings), and generates N questions via TestsetGenerator.
• Output schema: {"question": ..., "truth": ...} JSONL (evals/ground_truth.jsonl).
• Each truth string embeds a page-level citation token, e.g. ... [Northwind_Standard_Benefits_Details.pdf#page=7], parsed from the chunk's sourcepage.
• evals/evaluate.py consumes question / truth and its custom metrics (AnyCitationMetric, CitationsMatchedMetric, CITATION_REGEX) depend on those citation tokens being present in truth. Any replacement must preserve them.

Key challenge: recovering page-level citations

The Foundry simple_qna recipe emits only query / ground_truth rows with no source/page pointer, but our citation metrics need [Doc.pdf#page=N] tokens in truth.

Chosen design — group index chunks by page, one job per page

Control the upload granularity so that one uploaded document == one source page, giving every generated Q&A an exact page citation with no best-match approximation:

1. Read all chunks from the Azure AI Search index (id, content, sourcepage).
2. Group chunks by sourcepage (one PDF page of one file).
3. Reconstruct reading order within each page group by sorting on the integer suffix of the chunk id. Note: searchmanager.py:617 builds ids as f"{filename_to_id()}-page-{section_index + batch_index * MAX_BATCH_SIZE}" — that -page-N suffix is a global sequential chunk ordinal in document order, not the PDF page number (the real page lives in sourcepage). Sorting by it restores intra-page order.
4. Concatenate ordered chunk text into one per-page document (comfortably clears the 1KB upload floor that a single ~1000-char chunk would fail).
5. Sample a subset of pages (diversity/cost), upload each via files.create(purpose="user_data"), poll to processed.
6. Submit one SimpleQnA DataGenerationJob per page (max_samples >= 15 floor).
7. Map query -> question, ground_truth -> truth, and append that page's exact [Doc.pdf#page=N] citation to truth.
8. Aggregate across pages and downsample to the target count (e.g. 15 pages x 15 = 225 -> ~50).

This leaves evals/evaluate.py and the ground_truth.jsonl format untouched. Multimodal/figure citations (ground_truth_multimodal.jsonl) remain out of scope — they are hand-curated only.

Alternatives considered: whole-doc generation + AI Search best-match lookup (fewer jobs, but approximate citations) and single-chunk-per-document (exact but fails the 1KB floor and produces low-quality repetitive questions). Page-grouping was chosen as the best balance of exact citations, question quality, and job count.

Proposed upgrade path (phased)

Phase 1 — Infra: expose a Foundry project endpoint

• The SDK needs a Foundry (services.ai.azure.com) project endpoint https://<res>.services.ai.azure.com/api/projects/<proj> and azure-ai-projects>=2.2.0. Today useAiProject provisions the older hub-based AI project (infra/core/ai/ai-environment.bicep), and AZURE_AI_PROJECT is only the project name — not the endpoint the new SDK needs.
• Add infra to provision/point at a Foundry project and output AZURE_AI_PROJECT_ENDPOINT (full URL). Wire through infra/main.parameters.json, infra/main.bicep appEnvVariables, and both CI env sections (.azdo/pipelines/azure-dev.yml, .github/workflows/azure-dev.yml), per AGENTS.md.
• Grant the running principal the Foundry User role on the project.

Phase 2 — Rewrite generate_ground_truth.py

• Keep index-chunk retrieval; add group_chunks_by_page(...) (group by sourcepage, sort by id ordinal, concatenate).
• Add generate_ground_truth_foundry(...) using AIProjectClient: sample pages, upload, submit per-page SimpleQnA jobs, poll, resolve DatasetDataGenerationJobOutput, map fields, append exact page citation, downsample, write ground_truth.jsonl.
• CLI: --numquestions (target total), page-sample size, --groundtruthfile. Drop --kgfile (RAGAS-specific).

Phase 3 — Dependencies & cleanup

• evals/requirements.txt: add azure-ai-projects>=2.2.0; remove ragas, langchain, rapidfuzz.
• Delete evals/ground_truth_kg.json (28MB, RAGAS-only).

Phase 4 — Docs & tests

• Update docs/evaluation.md "Generate ground truth data" section: new command, options, region/model prerequisites, citation notes.
• Add unit tests mocking AIProjectClient / OpenAI Files API (mock at HTTP level per AGENTS.md).

Model requirement (gpt-5 upgrade)

Reuse the existing eval deployment (AZURE_OPENAI_EVAL_DEPLOYMENT) as the Foundry generator model, but upgrade it to a gpt-5 class model. The eval default is currently gpt-4.1 (infra/main.bicep:257) and the Foundry docs example uses gpt-4.1-mini, which is deprecated. Bump the eval model default to a gpt-5 model (e.g. gpt-5-mini, matching the chat model's move to gpt-5.4-mini) and confirm Responses API support (the simple_qna recipe requires a Responses-API-capable model).

Constraints / notes

• max_samples floor is 15 per job — hence the oversample-then-downsample approach.
• 1KB minimum upload size — page concatenation clears it; single chunks would not.
• Preview + region-limited. Synthetic data generation is supported in: UAE North, West US 3, North Central US, East US, West Europe, South Central US, Switzerland North, Sweden Central, East US 2, West US, France Central, South Africa North, Australia East, Japan East, UK South, Norway East, Poland Central, South India, Germany West Central, Italy North. (East US 2 and Sweden Central — the sample's common deploy regions — are both covered.)

Acceptance criteria

• evals/generate_ground_truth.py generates ground_truth.jsonl via the Foundry service with exact [Doc.pdf#page=N] citations preserved.
• evals/evaluate.py and ground_truth.jsonl format are unchanged; citation metrics still pass.
• RAGAS deps and ground_truth_kg.json removed.
• Infra outputs AZURE_AI_PROJECT_ENDPOINT; eval/generator model upgraded to gpt-5 class.
• docs/evaluation.md updated; unit tests added with mocked Foundry SDK.