A rollback plan tells you how to undo an AI change, but not what breaks first when the change stays in place.
Most production incidents do not begin with a deliberate rollback; they begin with an unexpected failure mode that the author never tested.
I now treat a passing fault drill as a precondition for reviewing any AI-generated server patch.
The drill runs on a disposable server before a human reads a single line of the diff.
Why a rollback plan is not enough A rollback plan answers a question about the past: how do we return the system to a known state?
A fault drill answers a question about the future: what happens when this change meets a condition the author did not imagine?
The second question decides whether you get paged at 3 a.m.
A change with a perfect rollback can still fail in a way that nobody notices until the data is gone.
Free model access changes the economics of this argument, because generation stops being the bottleneck and verification starts.
When a draft is nearly free, the cheapest verification is the one that breaks the change on purpose.
A rollback plan is documentation; a fault drill is evidence.
Documentation tells you what should happen, while evidence tells you what actually happens on a real service manager.
The fault drill in five steps The workflow assumes two cheap resources: a model that generates failure hypotheses from a diff, and a server that can be destroyed after the drill.
MonkeyCode's free model access covers the first, and its free server option covers the second, so a drill costs almost nothing.
Disclosure: This article was prepared as part of MonkeyCode's product outreach.
Any ephemeral VM or container host works if you prefer a different provider.
1.
Generate failure modes before you apply anything Ask the model to enumerate failure modes for the diff, and forbid it from proposing fixes, because fixes are a distraction at this stage.
The prompt below is the one I use, and it produces a catalog that the drill can test.
Treat the model's list as a hypothesis set, not a safety analysis.
The list is useful because it is cheap and broad, and it is dangerous only if you trust it without running the drill.
2.
Provision a disposable server and apply the change Provision a server that mirrors the target's OS and service manager, apply the change, and confirm that the service starts cleanly.
The server must be disposable, because the next step will deliberately break it and you should not care.
A shared staging box is the wrong target, since a drill that corrupts another team's state teaches the wrong lesson.
3.
Run the fault drill Run against the unit and a sentinel file that represents data the service must not lose.
The script applies each fault, waits a few seconds, records the service state, attempts a manual restart, and checks the sentinel.
Each fault is applied to a restored system, so one bad result does not contaminate the next row.
4.
Read the table like a reviewer Each output row classifies one failure mode along three axes: service state after the fault, recovery after a manual restart, and sentinel integrity.
A row with , , and is a reject, while a row with , , and is boring.
Boring is the highest compliment a failure mode can earn, because boring means the incident is already solved by existing machinery.
5.
Decide, and write the verdict into the review Approve the change only when every tested failure mode is boring or recoverable, and document the verdict next to the diff.
If any fault loses the sentinel, the change fails review regardless of how clean the diff looks.
A clean diff is a statement about intent, while a drill result is a statement about behavior, and behavior is what runs in production.
The artifact: fault-drill.sh The script is intentionally small, because a drill you cannot read in one sitting is a drill you will not run.
It takes a unit name, a sentinel path, and a list of faults, and it prints a classification table.
Run it only on the disposable server,