Built for the All Things Agentic Hackathon.
The project: an autonomous compliance agent for cannabis track-and-trace, running on Cloud Run with Gemini 3.5 Flash.
This post is about the test harness, because it turned out to be the most interesting part.
The objection every agent demo deserves Every hackathon agent demo has the same silent flaw: the person who built the agent also built the test.
Of course the agent passes.
The demo world was constructed, consciously or not, around what the agent is good at.
My project had this problem in an acute form.
It reconciles a cannabis facility's inventory against METRC - the state seed-to-sale system that is legally the record of what exists - and decides whether packages can be released under Kentucky's testing regulation, 915 KAR 1:110.
There's no way to demo against production METRC (vendor API access requires a training-and-agreement process), so the whole thing runs against a fixture service I wrote: a fake METRC that serves the documented v2 API shapes from Firestore.
So the pitch is "my agent correctly polices a world I invented." Rigged, by construction.
The fix: make the regulation the ground truth, then hand over the keys Two design moves converted the objection into the project's strongest property.
First: the agent is never the reference.
The regulation is.
The required test panels from 915 KAR 1:110 Section 2 are stored as structured data - ten analyte categories for finished product, three for an in-process batch, each with its citation.
Whether a package's testing is complete is a set difference computed in code, before any model is consulted.
Anyone holding the same METRC record and the same regulation text can check every decision.
Second: judges can construct worlds I never wrote.
The fixture service takes a compact spec: One POST expands that into a full world - realistic METRC package records, lab results with plausible levels, transfer manifests - served over the same API surface as the seeded demo.
Then you trigger a cycle and grade the agent yourself, against the regulation, on a case whose answer I never knew in advance.
What happened the first time someone (me) actually did this The first custom scenario ever run against the deployed system found three bugs in an evening.
None of them were the model being wrong.
All of them were the model being right about something I got wrong.
Bug 1: my "passing" data was physically absurd.
The generator gave every product the same passing values, so a concentrate carried 1.2% total THC and a solventless rosin carried a butane result.
Gemini escalated both as implausible - correctly.
A rosin with residual butane on file is what a wrong-matrix lab result actually looks like.
Lesson: generated test data must be boring.
If your fixtures trip the agent's plausibility instincts, every scenario drowns in false alarms and the agent looks paranoid instead of careful.
Bug 2: my records were internally contradictory.
A spec that failed a test still stamped the package in the METRC state field.
The model refused to act on the contradiction and asked for a human.
Right again.
The state-vs-results contradiction is a legitimate scenario - but it should be constructed on purpose, not seeded by accident.
Bug 3 (the one that mattered): escalation left a failing package movable.
My rules layer had a principle I was proud of: escalation performs no write, because an agent that escalates to a human and then acts anyway hasn't escalated.
The custom scenario exposed the flaw: the model escalated a failing-solvents record, no write happened, and a package with a failing required test stayed movable in the system of record.
A real compliance officer would never do that - you freeze the package pending review.
The fix distinguishes ambiguity (escalate, touch nothing) from hard evidence (a missing or failing required test: protective hold first, then escalate).
Freezing state isn't deciding the outcome; it's making sure nothing moves while the human