📖 TL;DR An MCP eval is a realistic task a model must complete using only your server's tools — not an assertion about one call.
A test asks "did the call work?".
An eval asks "could an agent get the right answer?" The signature failure is an answer that is wrong even though every call returned
200.
No inspector catches it.
There are four useful outcomes: pass, wrong answer, too many calls, and untestable.
The last one is not a failure.
Evals are not deterministic, and pretending otherwise is how you get a number you cannot trust.
Call count degrades before pass rate does, which makes it the earlier warning signal.
You built an MCP server.
The handshake works, every tool returns valid JSON, your integration tests are green.
Then someone connects it to Claude and it is useless.
That gap has a name now.
An MCP eval is what closes it.
The distinction is simple once you see it.
Your test suite proves the protocol works.
It says nothing about whether a model can use what you built.
Those are genuinely different questions, and only one of them is the reason your server exists.
The MCP specification has nothing to say about the second one, and it should not — conformance is not usability.
The 2026-07-28 release tightened the protocol considerably and did not change this at all.
I have spent the last few months building an eval engine for arbitrary MCP servers.
This post covers what an eval is, the four outcomes worth distinguishing, why the results are non-deterministic, and when you should bother.
What an MCP Eval Actually Is An MCP eval is a task, phrased the way a user would ask it, that a model has to complete using only your server's tools.
Not an assertion.
Not a mocked conversation.
A question, and a score for whether the model got there.
A test looks like this: An eval looks like this: Nothing in the eval names a tool.
Choosing the tool is the thing being tested.
The model gets your tool list, your descriptions and your schemas, and has to work out the rest.
That is exactly what happens in production.
MCP Eval vs Testing vs Inspecting These three get used interchangeably and they measure different things.
Answers Misses Inspecting Does the handshake succeed?
Do tools list?
Everything about usability Testing Does this call, with these args, return what I expect?
Whether a model would ever make that call Evaluating Can an agent reach the right answer from my descriptions?
Determinism, exhaustive coverage The key structural point: when you write a test, you have already made the choice the model has to make.
You picked the tool.
You picked the arguments.
You skipped the only step that can fail in the way that matters.
All three are worth having.
Testing catches protocol and correctness bugs, and it is faster and cheaper than an eval.
Evals catch a class of defect testing structurally cannot reach.
The Failure That Has No Red Log Line Here is the scenario that made me build this.
A docs server exposes and .
A user asks which regions a product supports.
The agent calls .
It gets back five ranked results.
It calls on the first one.
That page mentions two regions in an example snippet.
The agent answers: "It supports us-east and eu-west." Every call succeeded.
Valid JSON, no errors, no timeouts.
Your logs are clean and your dashboard is green.
The real answer was eleven regions, listed on a page the search ranked fourth.
This is the defect evals exist to find A wrong answer produced entirely from successful calls.
There is no error to catch, no exception to log, no status code to alert on.
The only way to detect it is to check the answer.
Nothing in your stack is lying.
The protocol worked perfectly.
The server was just hard to use, and the model did what it could.
The Four Outcomes and What Each One Means A binary pass/fail throws away most of the signal.
Four outcomes are worth distinguishing.
Pass The agent answered, stayed within its call budget, and the answer was correct.
This is the only outcome that requires judging content, which means it is the only one a model decides.
Wrong Answer The agent produced an answer and it was not right.
Either it could not find the information, or your tools gave it something misleading.
This is the outcome from the scenario above, and it is the most valuable one an eval produces.
Too Many Calls The agent got there, but it took eight calls when the budget was four.
This is the tool-description signal, and it is the one to watch over time.
The agent was guessing.
It tried a tool, got something unhelpful, tried another.
Every one of those calls cost you latency, tokens and — if your API is metered — money.
Call count degrades before pass rate does.
A task that passed in two calls last month and passes in six today is a regression, even though the number in the pass column did not move.
I covered the root cause of this in why 97% of MCP tool descriptions are broken.
Untestable Something outside your server broke.
The connection dropped, the driver model rate-limited, the harness fell over.
This is not a failure and it must never be counted as one.
It gets its own bucket for a reason I will come to.
See a finished eval run — tools detected, tasks written against them, a model made to complete each one, and a report showing where it went wrong.
No sign-up: Walk through MCP Evals → Why "Untestable" Deserves Its Own Outcome This is the design decision I would defend hardest, and it took me a while to get right.
A failure only means something if you know the input was valid.
Say an eval calls and gets a
404.
Is that a defect in your server?
It depends entirely on where that ID came from.
If returned it a second earlier, the ID demonstrably exists and your server cannot fetch it.
That is a confirmed defect, with both calls as evidence.
If a model invented it, a 404 proves nothing.
The issue probably just does not exist.
Same error code, opposite meaning.
The only thing that separates them is provenance.
This is why a serious eval engine harvests real values from your server before planning anything that needs them.
Failures on harvested values are defects.
Failures on invented values are untestable.
Collapsing those two into "fail" gives you a report full of noise that developers correctly learn to ignore.
Evals Are Not Deterministic — and That Is Fine Models write the tasks.
A model drives the tools.
A model grades the answer.
Run the same suite twice and you can get different numbers.
People find this disqualifying.
I think that reaction comes from expecting an eval to be a test, which it is not.
The research benchmarks live with the same constraint.
MCP-Atlas grades 1,000 tasks across 36 real servers using a rubric-driven judge, and MCP-Bench pairs rule-based checks with LLM scoring for exactly this reason.
Nobody has found a way to make this deterministic, because the thing being measured is not.
An eval is evidence about how an agent behaves against your server.
It is not a pass/fail certificate, and any tool that presents it as one is overselling.
Two rules make non-determinism workable: Reproduce before you act.
A single failure is a lead, not a finding.
Run it again.
A defect that reproduces is real; one that does not is variance.
Read the transcript, not the score.
The number tells you where to look.
The transcript — which tools were called, with what arguments, what came back — tells you whether the verdict was fair.
There is a third rule I only arrived at by getting it wrong: the grader must fail open.
If the judge model errors out or returns nothing for an item, that item stays passing.
A missing verdict is not evidence of a defect.
Failing closed would let an unrelated API hiccup invent bugs in your server.
When You Need Evals, and When You Do Not Evals cost model calls.
They are slower and pricier than tests.
Some servers do not need them.
Worth it when: More than about 8 tools Two or more tools that do similar things You are about to rewrite descriptions You ship to users you cannot observe You support multip