Amazon Bedrock AgentCore Evaluations solves a real problem: you build agents on LangGraph, your teammate uses LlamaIndex, and the platform team is experimenting with the OpenAI Agents SDK.
Every framework has its own evaluation story, and none of them talk to each other.
AWS's answer is to treat OpenTelemetry telemetry as the evaluation contract.
If your agent emits the right spans and attributes, AgentCore can score it without knowing which framework you used.
The service works with LangGraph, LlamaIndex, OpenAI Agents SDK, Google ADK, Claude Agent SDK, and Strands Agents.
It also works with custom stacks, as long as you instrument them correctly.
This is the first major cloud vendor to decouple agent evaluation from framework choice using a telemetry-based contract.
Here's how the plumbing works.
The OpenTelemetry Contract AgentCore Evaluations expects agents to emit structured telemetry in OpenTelemetry format.
The service looks for specific span types and attributes: Agent spans: Top-level execution context for a single agent invocation Tool call spans: Individual tool invocations, including input, output, and latency LLM spans: Model calls with prompt, response, token counts, and model ID Retrieval spans: Vector search or knowledge base queries Each span must include semantic attributes that map to evaluation dimensions.
For example: The key insight is that AgentCore doesn't care about your framework's internal state machine or graph structure.
It only cares about the observable events: what tools were called, what the LLM said, how long things took, and whether they succeeded.
Framework Adapters and Instrumentation Gaps Most popular frameworks already emit some OpenTelemetry telemetry, but the coverage varies: Framework Native OTel Support Instrumentation Gap Workaround LangGraph Partial (LangSmith integration) Missing tool success/failure attributes Manual span enrichment LlamaIndex Good (built-in OTel exporter) Inconsistent span naming Span processor to normalize OpenAI Agents SDK Minimal No tool spans by default Wrap tool calls with custom tracer Claude Agent SDK None Everything Full manual instrumentation Custom stacks None Everything Build from scratch If your framework doesn't emit the required telemetry, you have three options: Manual instrumentation: Wrap your agent code with OpenTelemetry API calls Auto-instrumentation: Use OpenTelemetry's auto-instrumentation libraries for HTTP, database, and LLM calls Span processors: Intercept and enrich spans after they're created but before they're exported AWS doesn't provide framework-specific shims.
You're responsible for making sure your agent emits the right telemetry shape.
The documentation includes example instrumentation for each supported framework, but you'll need to adapt it to your specific agent architecture.
Evaluation Metrics and Scoring Once AgentCore receives telemetry, it computes metrics across several dimensions: Task success rate: Percentage of agent runs that completed without errors Tool accuracy: Whether the agent called the right tools in the right order Response quality: LLM-as-judge scoring of final outputs against ground truth Latency: P50, P95, and P99 for agent runs, tool calls, and LLM calls Cost: Token usage and estimated inference cost per run The service stores evaluation results in a time-series database (likely Amazon Timestream, though AWS doesn't specify).
You can query results via the AgentCore API or view them in the AWS console.
Retention is 90 days by default.
After that, you need to export results to S3 if you want long-term storage.
Deployment Shape and Data Flow Here's the typical data flow: Your agent runs in your AWS account (Lambda, ECS, EC2, or on-prem) The OpenTelemetry SDK batches spans and exports them to the AWS Distro for OpenTelemetry (ADOT) Collector The ADOT Collector forwards spans to AgentCore Evaluations via AWS PrivateLink AgentCore processes spans, computes metrics, and stores results You query results via the A