There is a problem with most benchmarks for coding agents: they are not software development.
They are useful, of course.
Give an agent an issue, run a test suite, check whether the patch passes.
SWE-bench and similar evaluations give us a standardized way of comparing models.
But this is not how I actually use agents.
I don't want an agent to fix a single isolated issue and stop.
I want it to spend hours inside a codebase, understand the architecture, write requirements, implement things, run tests, benchmark alternatives, discover that its first idea was wrong, revise it, document what happened, and then use that new knowledge in the next task.
So, during the last few days, I accidentally created another kind of benchmark.
I gave DeepSeek Harness (DSH) + DeepSeek V4 Pro a real project and let it work.
The result was: Metric Result Visible development window Aug 31 → Sep 3 Git commits 45 Files changed 193 API requests 3,266 Tokens processed 612,923,681 Cache-hit input 600,885,248 Cache-miss input 6,458,427 Output 5,580,006 Input cache-hit rate 98.94% API cost $28.35 Yes. 612 million tokens.
Twenty-eight dollars.
And before anyone starts typing "tokens are not productivity" into the comments: I agree.
That is exactly why the interesting part of this experiment is not the token count.
It is what happened to the repository.
The Project The project is LovelaceSharp, my attempt to build an arbitrary-precision mathematical environment in C#.
It contains arbitrary-precision natural numbers, integers and real numbers, a scripting language, vectors and N-dimensional arrays, linear algebra, a web IDE, benchmarking tools and a Lean project for formally verifying the underlying arithmetic model.
This is a useful agentic workload because it is not a CRUD application.
There are plenty of opportunities for code that looks correct but isn't.
Arithmetic has edge cases.
Numeric algorithms have crossover points.
Performance changes can destroy correctness.
Array views introduce aliasing and stride semantics.
Precision can leak across sessions.
Formal proofs can prove something subtly different from what the optimized production implementation actually does.
In other words: bullshit has somewhere to hide.
That makes it a much more interesting test than asking an agent to add another REST endpoint.
Establishing the Boundary I also got lucky with the Git history.
The old work on LovelaceSharp stopped on March 17 at commit .
There was then no activity until August 31, when the new DSH-driven development started.
The comparison from that old baseline to the end of the visible burst contains: 45 commits across 193 changed files.
You can inspect the history and comparison directly: LovelaceSharp commit history Baseline-to-current comparison So I am not counting the previous implementation as work performed by DSH.
That matters.
The benchmark isn't: "Look at this entire repository an AI supposedly made." It is: "Here is the repository before this run, here is the repository after it, and here is the API bill during the days those changes landed." My complete DeepSeek export spans Aug 30 through Sep 5 and contains roughly: 805.4M tokens 3,931 requests $36.68 of spending For the benchmark, however, I only attribute Aug 31 through Sep 3, because those are the dates represented by the visible development burst on .
That leaves: 612.9M tokens and $28.35 associated with the 45-commit comparison.
So what did those $28 actually buy?
First: Replace the Core Number Representation One of the largest changes was rewriting , the arbitrary-precision unsigned integer implementation.
The previous representation was based around decimal BCD storage.
The new implementation moved the actual number representation to little-endian base-2^64 limbs.
That is already a reasonably dangerous refactor because essentially every numerical type above it depends on .
The change included: native carry/borrow arithmetic; schoolbook multiplication; Karatsuba multiplication; Knuth Algorithm D division; short division; divide-and-conquer conversion between decimal and binary representations.
But the important part is what came with it.
DSH added tests specifically around limb boundaries and randomized differential testing against .
The differential suite covers arithmetic across multiple operand sizes and adversarial values around boundaries such as and .
Relevant commit: Rewrite Natural storage around 64-bit limbs That is much more interesting to me than "the model knew Knuth division." LLMs know algorithms.
The useful behavior is: That is engineering.
Then It Made Multiplication More Complicated — for a Reason After moving to binary limbs, the agent added an exact two-prime Number Theoretic Transform multiplication path for very large numbers.
This wasn't simply: NTT is asymptotically fast, therefore NTT everywhere.
It benchmarked the implementation against Karatsuba and introduced a dispatch threshold around 100,000 combined limbs, roughly the region where the NTT path actually started winning.
It also cross-checked multiplication against at extremely large operand sizes.
Relevant commit: Add NTT multiplication for huge Natural values This distinction matters enormously when evaluating agents.
A coding model can produce complicated code all day long.
A useful engineering agent must understand that an asymptotically superior algorithm can still be the wrong implementation for almost every practical input.
And that became even more obvious with division.
Then It Implemented an Algorithm and Discovered It Should Barely Use It DSH implemented Newton-reciprocal division with asymptotically better behavior for huge operands.
Then it benchmarked it.
And Knuth division won.
Not forever, but for a surprisingly long time.
The measured crossover put Newton at roughly the multi-million-decimal-digit range, so the production dispatcher retained Knuth below that region and only selected Newton for enormous inputs.
Relevant commit: Add Newton reciprocal division and benchmark its crossover This may be my favorite part of the experiment.
Because the workflow was essentially: I want more agent benchmarks to measure this.
Not whether the model knew Newton iteration.
Whether the agent was willing to prove itself wrong.
The Work Wasn't Limited to Big Integers During the same development burst, Lovelace gained a typed-array abstraction with concepts such as: dtype metadata precision metadata slices strided views a plugin-oriented kernel contract The abstraction arrived with tests covering array layout and view behavior.
Relevant commit: Introduce typed array abstractions Then the scripting engine itself was migrated onto the typed representation, adding things such as: promotion; narrowing; broadcasting; slicing; views; empty-dimension behavior.
Relevant commit: Migrate Suite to the typed array representation Around the same period, the project also acquired a web IDE, , over the common scripting engine.
The initial Studio work extracted the scripting language into a shared engine and built an ASP.NET Core + browser IDE around it.
Relevant commit: Introduce Lovelace.Studio A later change added isolated sessions, per-session precision, incremental computation, asynchronous progress and CodeMirror autocomplete.
Relevant commit: Add isolated sessions and incremental evaluation The point is not LOC.
The point is that the workload moved through: numerical algorithms, language implementation, web tooling, concurrency, arrays, benchmarks and formal methods without changing the overall agentic workflow or treating each category as a fresh synthetic benchmark.
That is much closer to the way I want an engineering agent to behave.
It Also Started Proving Things Another part of the burst introduced , a Lean 4 project formalizing base-b arithmetic.
It covers representation, addition, subtraction, multiplication and division using core Lean.
Relevant commit: Add Lean proofs for core arithmetic There is an important caveat here.
The Lean proo