Four bugs in one paragraph: what I learned building an autonomous loop that chases people who don't answer

2026年8月30日1 次浏览来源:Dev.to阅读原文

I built CrisisMesh for the All Things Agentic Hackathon, and I created this post for the purposes of entering that hackathon. #AllThingsAgenticHackathon CrisisMesh is a seven-agent fleet on Google ADK and Gemini 3.5 Flash that coordinates a school's response after a human reports an emergency.

It doesn't detect anything.

Somebody sends the message they'd already have sent: a Slack command, a WhatsApp text.

The fleet takes it from there.

The interesting part wasn't getting agents to talk to each other.

It was everything that happened once the system started making claims.

The loop is the product The feature I'd defend is the one nobody watches.

A declared incident starts a scheduler.

Every tick it looks at whoever hasn't checked in, pings them, re-pings them, and at a configured cap stops pinging and hands that person to their floor warden by name, on the warden's own channel.

Four bugs lived in that one paragraph.

Every one of them passed its tests.

It sent the escalation to the person it couldn't find. "Mrs.

Rodriguez has not answered, please locate her" went to Mrs.

Rodriguez.

The message was correct.

The recipient was the one person it couldn't help.

It never stopped.

Once escalated, a person stayed actionable, so on a timer the same warden was paged about the same person every 25 seconds, forever.

I fixed it with a terminal state, and then the same bug came back, because the guard went into the function the tests called while the running loop called a different one.

Green the entire time.

It counted people it couldn't reach as reachable.

Any non-empty Slack id counted, including the roster's placeholders like , so the loop reported 34 of 34 reachable and chased thirty people down channels that addressed nobody.

It now verifies ids against the workspace and reports 4 of

34.

That number is worse and it's true.

The difference matters: unreachable is a fact a commander acts on, not a gap to paper over.

The tick guard was process-local.

Harmless at one instance.

At four, every container runs its own tick N and one silent teacher gets pinged four times.

Ticks are now claimed with a create-if-absent lease in Firestore.

Every real bug was in a seam Not one of these was a bad function.

They were all connections: a guard in the wrong function, a check-in written to one ledger and not the other, a critic that ran on one path and not another.

The sharpest example: a teacher asked "what's the fastest route out of east wing" during an active shooter, and the system answered with corridor directions.

The movement policy that exists to prevent exactly that ran inside the fan-out.

A query answer is a transport reply, and it never passed through.

Unit tests for the policy: green.

Unit tests for the query desk: green.

So the tests changed shape.

They stopped asserting that functions return values and started asserting things about the world: Those fail for real reasons.

Managed services find bugs that mocks cannot Moving the Memory Bank onto Vertex AI Agent Engine took an afternoon of writing and ten minutes of reality demolishing it.

Vertex persists and and silently drops and , which is where I'd put the structured record, so retrieval returned memories it couldn't read.

And scope matching is exact on the whole map, not a subset, so putting metadata in scope makes a lesson findable only by someone who already knows its id.

Neither is in the docs.

Both took one live call.

I made the test double lossy in the same way afterwards, so the double can no longer pass while the service fails.

Model Armor was worse, and the failure was mine.

The pillar was marked managed while the deployed service ran the regex fallback, so none of it had ever executed in production.

Three defects were sitting there: The client was built against the global endpoint.

Templates are regional, so every scan returned "template not found", which fell into the error path below.

Both error paths returned , under a comment claiming they failed closed for ambiguous cases.

The

分享
Baike.dev

baike.dev helps you discover great languages, frameworks, databases, DevOps and cloud-native tools.

Quick links

About

Contribute

Found a great developer tool? Share it with the community.

Submit a tool
© 2026 baike.dev Developer EncyclopediaUpdated daily · Discover great developer tools