[evaluation] Diagnose GPT-5.3 Codex stopping after skill activation

Problem

Current GPT-5.3 Codex evaluation traces frequently load the requested skill and then immediately return a final response without inspecting, editing, or validating the workspace.

In scheduled run https://github.com/dotnet/skills/actions/runs/34680113000, 12 of 18 isolated code-testing-agent trajectories performed exactly one tool call—the skill invocation—then stopped. The equivalent GPT-5.6 Luna run had 0 of 18 such trajectories. GPT-5.3 averaged 11.9 tool calls in the baseline arm but only 3.9 in the skilled arm for this eval.

A representative migration trace loaded guidance that explicitly said “Skill activation is not completion,” “inspect the staged files,” and “do not ask the user for a discoverable path,” then emitted phase: final_answer asking the user to provide files without making another tool call.

Why this may be infrastructure-level

• The runtime injects the loaded SKILL.md as a new user-role <skill-context> message.
• A second model request is made, so this is not simply a missing host loop iteration.
• OpenAI's Codex guidance requires preservation of assistant phase, reasoning items, and tool-call/result ordering; losing those can materially degrade continuation.
• The evaluation matrix selects the model but does not pin reasoning effort or context tier.
• The evidence was produced by Copilot runtime 1.0.80, while newer runtime versions exist.

Proposed investigation

Run the same implementation-heavy stimuli under a controlled factorial:

1. baseline, automatic activation, explicit skill invocation, and the same skill body pasted directly into the prompt;
2. current pinned runtime versus the latest supported runtime;
3. low, medium, and high reasoning effort;
4. isolated skill versus full plugin;
5. primary and independent secondary judges, while prioritizing deterministic completion graders.

Add trace telemetry for:

• skill selected and body injected;
• first non-skill action after activation;
• activation-only terminal responses;
• finish reason and assistant phase;
• preserved reasoning/tool-call items;
• workspace reads, writes, validation commands, and deterministic completion.

Acceptance criteria

• The harness can distinguish routing failure, post-activation continuation failure, tool failure, and content-quality failure.
• Implementation prompts that stop after only loading a skill are surfaced explicitly rather than appearing merely as low-quality output.
• Model, runtime, reasoning effort, context tier, and tool-schema identity are recorded with every result.
• A controlled reproduction determines whether the behavior belongs to GPT-5.3, the Copilot provider adapter, the skill-injection transcript, or skill wording.