Last week I turned my data audit into a build step: a check that runs before anything else and fails the build when the database and any static copy of my travel site's legal-status data disagree.
It ended the era of the site contradicting itself.
It did nothing about the site agreeing with itself on something false.
That's not a hypothetical.
The most expensive error the whole project found was a country whose law changed in January while every copy on my site — database, data files, search index — kept saying the old thing in perfect unison.
Internal consistency was the camouflage.
No diff between my own sources could ever have caught it, because every internal source was equally behind the world.
A build gate proves agreement.
Agreement is not truth.
Something has to look outside.
You can't diff against the world, but you can sample it The naive version of "look outside" is another audit — a human session checking primary sources jurisdiction by jurisdiction.
I've done three of those now, and I know exactly what they're worth: they're correct the day they ship and they decay from that morning on.
Laws don't change on my audit schedule.
So the outside check became what the inside check became: a scheduled job.
Once a week, a script asks a web-connected model — one that searches and cites, not one answering from training memory — for the current legal status of about fourteen jurisdictions, and compares each answer to the corresponding database row.
Fourteen, not all 271, because the selection is doing the real work: A hot list is checked every single run: the highest-traffic pages plus the jurisdictions with active legislative motion — the places where being a month stale costs the most.
Everything else sits on a rotating cursor: eight per run, round-robin, so every row on the site gets sampled roughly twice a year without any run costing more than a few cents.
The whole thing runs on about seven cents a week.
Two rules were non-negotiable, both inherited from the build gate: It never writes.
A model's answer is not data — it's a tripwire.
When the model and the database disagree, the script sends me a flag that says, in so many words: verify this against primary sources, do not trust it.
Every fabricated-but-plausible fact I've written about in this series was an AI-generated value that skipped exactly this step.
The pipeline is model → flag → human → primary sources → fix.
The model never touches the last three.
A run that can't check anything is a failure, not a quiet pass.
If the database is unreachable or every query errors, the job exits nonzero and my monitoring treats it like any other broken cron. "Couldn't verify" reported as success is the most expensive bug a verification step can have — same rule, new layer.
Each flag also alerts exactly once per claimed value.
A definitional disagreement I've decided not to act on shouldn't renag me weekly; an alert channel that repeats itself gets muted, and a muted tripwire is decoration.
First run: four alerts, three of them mine The first live run checked thirteen jurisdictions and raised four flags.
Three of them said the same alarming thing: states my database grades as medical — real, functioning patient programs — were, according to the model, illegal.
All three flags were wrong.
And the model hadn't failed.
My question had.
I'd phrased the prompt the way I think about the site: "what does an ordinary adult visitor face here?" For a medical-only state, the honest answer to that question is prohibition — a visitor has no local patient card and no way to get one.
The model answered the question I asked, correctly.
My database answers a different question: what is the jurisdiction's regime.
Medical program: exists.
Same territory, same statutes, two defensible grades — because grading needs a rubric, and my dataset and my prompt were using different ones.
This is the part I'd generalize furthest: a comparison is only as strong as the rubric both sides share.
Last month I r