Website benchmark caption says medians, but source report uses means
Summary
The benchmark caption on https://ponytail.dev/#numbers calls the displayed results medians, while the repository's report describes the same results as means. This makes the published aggregation method inconsistent even though the headline percentages agree.
Reproduction
Observed on 2026-09-04:
- Open the official homepage's benchmarks section. It displays 54% less code, 22% fewer tokens, 20% lower cost, and 27% faster, with the caption
medians across 12 feature tasks on a FastAPI + React repo. - Compare the agentic report's summary: it describes the aggregation as a mean across each tier's tasks, with every task averaged over four runs, and lists the same four percentages.
- The README also explicitly describes the 54% figure as a mean; merged #165 clarified the distinction between that average and the per-task peak.
Expected / actual
Expected: the website uses the same aggregation label as the linked repository methodology (mean/average), or links a separate median-based analysis if that is the intended source.
Actual: the live caption says median while the repository's source report says mean. These statistics are not interchangeable, so readers cannot tell which methodology the site is advertising.
This report concerns the caption, not a challenge to the measured percentages. No model/API benchmark was rerun.
Scope
I could not find the deployed website source among the 159 tracked files at current main 974d940a1c5344210874150b98ff0d2c861fab6a, so there is no safe in-repository patch to attach. The source report should not be changed merely to match the website. Existing #454 establishes this repository as a place for official-site corrections. Searches for ponytail.dev, median, medians, and mean found no equivalent caption report.
Source: DietrichGebert/ponytail