Here is a Python function and a test for it.
That test gives you 47% line coverage.
It gives you a 9.5% mutation kill score.
The harness generates 21 small breakages of that module.
The suite notices 2 of them.
That gap is the whole reason I started this project.
Line coverage is the default test-quality signal in most of the industry, and it measures whether a line ran.
It does not measure whether anything would have complained if the line were wrong.
AI-generated tests are unusually good at producing that shape: high coverage, low detection.
Mutation testing measures the real thing.
You break the code in small ways and check whether the tests notice.
If no test fails, that is a bug your suite cannot detect.
It has been sitting there the whole time.
Mutation testing never went mainstream, and I think the reason is simple.
It hands you a wall of surviving mutants and no path to fixing any of them.
It tells you that you have a problem and then leaves.
So the idea was an agent that closes the loop.
Find the survivors.
Write tests that kill them.
Gate each generated test on a hard criterion: keep it only if it passes on clean code and fails on the mutant.
Ground truth is a subprocess exit code.
No model judges any outcome.
I built it over about 30 hours for the micro1 Frontier Engineering Challenge, which had around 7,800 registrants.
The tool works, sort of.
It is incomplete and I will get to the numbers.
But that is not the interesting part of the weekend.
The interesting part is that my measuring instrument kept lying to me, and it lied in a consistent direction.
The finding that broke my own premise Before I ran a single agent call, I ran the harness across 12 widely-used, well-maintained Python libraries: cachetools, validators, natsort, dictdiffer, toolz, voluptuous, python-slugify, python-dotenv, shortuuid, boltons, aiofiles, tenacity. 455 mutants generated. 133 survived the existing test suites.
Then I checked something I had assumed I would not need to check.
Of those 133 survivors, how many sit on a line the tests actually execute?
53.
The rest were never run at all.
Not weakly tested.
Not vacuously tested.
Untested.
I suspected my test commands were scoped too narrowly, so I widened them per target, between 6 and 40 times more test code.
If the "executes but does not assert" category was real and I was just missing it, that number should climb.
It went from 54 to
53.
Down.
And where widening changed anything, it converted unreachable mutations directly into kills.
It did not move them into the middle category.
It skipped it.
So in mature, human-written Python, the vacuous test failure mode is rare.
Where these suites fail, they fail by not running the code at all.
The story I had absorbed about tests that execute everything and assert nothing is a story about AI-generated tests.
It is not a story about human ones.
I had to reframe the project before I had built the main part of it.
That was the first sign that what I was actually building was a measuring instrument, and that I had not been treating it like one.
Eight bugs in the instrument Every one of these would have produced a confident, wrong, publishable number.
Editable installs made mutations invisible. on src-layout packages resolves imports back to the original checkout.
My mutations were written to a temp copy, so they never executed.
Three targets silently scored 0.000.
That would have read as "the agent fails on src-layout packages," which is a finding.
It is just not a true one.
Concurrency corrupted one target.
Running mutants in parallel gave me three different survivor sets across four runs, on the one target doing real async I/O.
I had already drafted a result of "0.27 to 0.77" off that.
It was noise.
Note which way it pointed: spurious failures get counted as kills, and kills are the number every arm is trying to increase.
A file picker chose the wrong test file.
On the hardest target.
Which means the model would have been shown irrelevant context in