[online evals] [evals][agents] give pxi more detailed skills/instructions for creating good evaluators
Author: ehuttCreated Jul 24, 2026Updated Sep 18, 2026
Labelsenhancementc/evals
Summary
Give PXI a shared evaluator-authoring skill that teaches it how to reason about and build high-quality evaluators across UI, MCP, and coding-agent surfaces.
Context
Trace review showed that PXI often writes an overly simple evaluator before defining what the evaluator is intended to capture. Quality improved when it was explicitly prompted to enumerate criteria first.
Proposed workflow
- Define the construct being evaluated and its boundaries.
- Enumerate success, failure, partial-credit, abstention, and edge-case criteria.
- Select an appropriate evaluator type and output schema.
- Identify representative calibration cases.
- Implement and preview the evaluator.
- Compare results against the criteria and revise before persistence.
Acceptance criteria
- Evaluator creation includes an explicit planning step before code or prompt generation.
- The guidance covers code evaluators and LLM judges.
- The skill is shared across applicable Pixie, MCP, and coding-agent surfaces rather than duplicated in divergent prompts.
- The workflow includes representative positive, negative, boundary, and malformed cases where relevant.
- Evaluator creation traces make the planning and validation steps observable.
- Regression evaluations measure whether the skill improves evaluator quality.
- PXI can discover and interact with evaluators associated with a project while preserving the relevant project context.
- PXI can appropriately use the evaluator playground to create, edit, and preview evaluators, distinguishing playground work from persisted project evaluators.
Considerations
- Define how PXI should select project evaluators and transition between project context and the evaluator playground.
Part of #14725.
Source: Arize-ai/phoenix