`OnlineEvaluation` exports prompts via evaluation explanation logs regardless of `include_content`

Summary

OnlineEvaluation emits evaluator explanations and failure messages as OpenTelemetry log records, with no reference to the agent's include_content setting. An evaluator that quotes the prompt or the output — which is what explanations are for — therefore exports that content even when the agent under evaluation has content capture disabled.

Detail

pydantic_evals/_otel_emit.py:

if result.reason is not None:
    attrs[_ATTR_EXPLANATION] = result.reason
_get_logger().emit(LogRecord(event_name=_EVENT_NAME, body=_format_result_body(result), attributes=attrs))

and the failure path does the same with failure.error_message. Both land as gen_ai.evaluation.explanation. Evaluation emission also copies baggage by default.

An agent configured with InstrumentationSettings(include_content=False) plus OnlineEvaluation exports its prompt this way, through an evaluator's EvaluationReason.reason.

Why this matters separately

This is not a regression — it is independently-enabled evaluation telemetry with its own purpose, and an explanation that can't quote what it judged is much less useful. But it does mean include_content=False is not a whole-trace guarantee once online evaluation is on, which is how users are likely to read it. It also can't be caught by span-level checks: tests/test_instrumentation_content_redaction.py scans spans, and this is a log.

Suggested direction

Decide the policy explicitly rather than by omission. Options: honour the agent's include_content for explanations; give evaluation emission its own setting; or document that evaluation telemetry is outside the include_content boundary. Whichever it is, it's worth saying somewhere a user will find it, since the natural reading of include_content=False is "no content leaves this process".

Found by a review pass over the GHSA-4x9p fixes; reproduced end to end with an agent plus OnlineEvaluation.