[Feature] Add representative cost/quality evaluations for OMX’s Astra defaults

Problem

Merged PR #3622 changed OMX's default leader, specialist, fast-agent, low-complexity worker, and Sparkshell models to Astra while preserving reasoning-effort defaults and explicit overrides. The PR explicitly states that its CI validates routing and compatibility, and that it includes no comparative OMX task benchmarks for output quality, latency, or cost.

Add representative task evaluations for that upstream default-policy change. The question is where Astra's additional capability improves completed OMX work, and where a supported alternative configuration achieves comparable quality with less usage or latency. This is an evaluation request, not a claim that the defaults are defective.

Proposed solution

Create a small, repeatable evaluation set covering the OMX roles and execution paths affected by #3622:

OMX surface	Representative task and quality check
explore / lightweight agents	Locate a behavior and its relevant files; check accuracy, coverage, and unnecessary follow-up work.
executor	Implement a bounded change in a fixed fixture repository; run its required checks and assess scope/correctness.
code-reviewer	Review fixtures containing known defects and clean controls; record missed defects and unsupported findings.
Low-complexity Team workers	Complete a bounded delegated task; check the handoff and integrated result, including any leader repair.
Sparkshell summaries	Summarize fixed command output; check preservation of failures and decision-relevant evidence.

Compare OMX's Astra defaults with explicitly recorded, supported model overrides on the same fixtures. Keep task requirements, workflow stages, tools, and per-role reasoning effort fixed for the initial model comparisons. Use a pinned OMX revision and an isolated configuration so the baseline does not inherit a contributor's personal settings; verify the effective model for each evaluated role.

Report completion quality, elapsed time, available usage, and retries or escalation through the final result. Include leader and child work where attribution is available. Mark missing usage as unknown and avoid a complete cost claim when attribution is incomplete. A lower token count alone is not a successful result.

Acceptance criteria

• Reproducible fixtures exercise the affected OMX roles/paths, reusing existing evaluation coverage where suitable. Keep task fixtures and quality checks separate from model configuration so they remain reusable for future OMX model changes; Astra is the initial baseline.
• Include a deterministic, no-model baseline for OMX tasks whose output contract permits it, such as exact status extraction or short command-output handling; retain interpretation and required review where needed.
• The baseline and each supported override are pinned and inspectable, including effective model, reasoning effort, service tier, and observable cache conditions. Mark unavailable runtime details as unknown; do not assume OMX controls Codex-owned caching.
• Results include correctness, required-check outcomes, review misses/false positives, retries, latency, available usage, and operator intervention/review/repair time. Report human effort separately so reduced model usage does not hide work transferred to the operator. Report difficult and failed cases separately. Include workload-weighted results only with documented, representative task frequencies; otherwise report per-task results without claiming representative savings.
• Where attribution is supported, distinguish task execution from delegation, repeated discovery, handoff repair, and continuation after completion. Classify coordination as avoidable only with evidence that required work and quality are preserved; leave ambiguous attribution explicit.
• Separate initial fixed-effort model comparisons from any later supported model/effort tuning against the same predeclared quality requirements. Identical effort labels do not establish equivalent quality or cost.
• Required workflow stages and independent review remain intact in all applicable configurations.
• An evidence-backed report ranks OMX-specific recommendations by observed benefit, quality risk, and estimated implementation/maintenance effort, including added routing rules, configuration, and regression coverage. Label estimates separately from measurements and explain whether the expected benefit justifies the added complexity. Inconclusive results or retaining the current defaults are valid outcomes.

Related work and scope

• #3622 — direct basis: merged Astra-default policy change and its explicit comparative-evaluation gap.
• #3624 — release context: tracks delivery of those defaults; this evaluation is not an additional release gate.
• #1714 / #1719 — reusable evaluation work: the specialist routing/quality issue was resolved by merged routing regression coverage. Inspect and reuse suitable fixtures; routing correctness alone does not measure end-to-end model quality or cost.
• #3226 — separate accounting proposal: closed without implementation pending owner-approved accounting and budget semantics. This issue uses available telemetry; it does not reopen canonical billing-ledger or budget-enforcement work.
• #3149 / #3180 — historical cost-related fix: the native-child continuation issue was addressed by a narrow Stop auto-nudge repair. Keep that fix intact and distinguish OMX behavior from Codex-owned context/runtime controls; this evaluation does not assume the repaired defect has returned.

Completion consists of reusable evaluation fixtures and the evidence-backed report. Implementing recommended optimizations belongs in separate follow-up issues. Default changes, automatic model/effort routing, accounting or budget infrastructure, broad prompt/orchestration redesign, and changes to Codex-owned context, caching, or compaction controls are outside this issue's initial scope.