Evaluation gaps after the skill-validator to Vally migration

Problem

Inspired by https://github.com/dotnet/skills/pull/1165 by @Evangelink - I realized we had that functionality on skill-validator. So I run copilot audit of functionalities that existed on skill-validator that we potentialy lost during move to Vally.

Area	What changed	Why it matters	Reviewed	Needs fix
Tool-use checks stopped working	The current corpus has 47 expect_tools and 73 reject_tools declarations, but Vally does not turn them into executable checks.	An evaluation can pass even when the agent skips required tools or performs a forbidden edit. These are grading rules, not security restrictions.	☐	☐
“Whole plugin” is not the actual whole plugin	The MSBuild treatment loads 18 skills, but not its three manifest agents or binlog MCP server.	Tests can exercise fallback behavior instead of the production agent/MCP workflow.	☐	☐
Required helper skills were dropped	Eight dependency declarations were removed from run-tests, including filter-syntax and platform-detection.	Isolated evaluations omit helpers that the skill explicitly expects to use.	☐	☐
Regex checks changed meaning	Native regex checks ignored case; Vally defaults to case-sensitive matching. 542 otherwise unchanged migrated patterns lack an explicit ignore-case flag.	Capitalization alone can change a grade. I reproduced this with unused versus Unused; the count does not mean all 542 currently fail.	☐	☐
Some substantive scenarios disappeared	C# scripting lost its language-agnostic nonactivation scenario and its executable multi-file #:include / #:exclude scenario. Both remain absent.	Advertised behavior lost coverage, including a test proving that invalid excluded source really stays excluded.	☐	☐
Timeout and retry semantics weakened	One build grader formerly inherited 600 seconds but now defaults to 30 seconds. The experiment runner also omits retries for otherwise retry-eligible rate-limit failures.	Valid slower builds can fail, and transient errors can terminate evaluations sooner. The budget change is proven; its production failure frequency is not.	☐	☐
Local evaluation no longer mirrors CI	The documented wrapper uses an unpinned Vally command and omits CI’s generated plugin treatment, expected-result accounting, and several hardening steps.	Contributors can run a materially different experiment. An offline probe also showed the actual wrapper returning success after a partial harness failure.	☐	☐
Summary results lost useful evidence	Adapted results omit richer assertion/tool details, judge-token usage, and some per-arm timeout information.	Judge usage can appear as zero rather than unknown; plugin timeouts can disappear from the summary. Much of the richer evidence still exists in raw artifacts.	☐	☐
Evaluation selection has blind spots	Changes to the retained native overfitting judge, toolchain lock, and local wrapper do not select evaluation through the current infrastructure filters.	Parts of the active evaluation system can change without exercising the evaluation path. Separate smoke/unit tests are not equivalent coverage.	☐	☐

Suggested next steps

We should go through those and validate which are real gaps that really matter to us. Then create subitems and fix them