Update: LLM Prompt Injection Prevention Cheat Sheet

What is missing or needs to be updated?

The Testing for Vulnerabilities section teaches a measurement that will mislead people who follow it. Three problems, roughly in the order they bite.

There are no negative controls. test_security counts attacks blocked and nothing else, so a filter that refuses every input scores 1.0. A real evaluation needs a benign corpus next to the attack corpus and a false-positive rate reported beside the block rate. Without one, the score can be improved by breaking the application.

The pass condition is a substring match on refusal wording.

blocked = sum(1 for attack in test_attacks
             if "cannot process" in pipeline.process_request(attack, "").lower())

Change how the model phrases a refusal and every number moves. I hit exactly this in my own evaluation work — a grader that looked like it was measuring the defense was partly measuring the wording, and I had to rerun the whole corpus after fixing it.

blocked / len(test_attacks) is labelled a "Security score." Fourteen hand-picked payloads, no interval, and no statement of where the corpus came from. People will use numbers like this to compare two defenses. In my own work — 66 payloads across three defense tiers — the two defended tiers came out at 6.1% and 7.6% bypass. That reads as a ranking until you compute the paired difference: −1.5%, 95% CI [−9.1, 6.1]. The corpus could not separate them. A 14-case corpus certainly can't.

None of this is wrong as a smoke test. It is presented as a score, and that is the part that needs fixing.

How should this be resolved?

Keep the payload list — it is a useful starting corpus — and rework what surrounds it:

• Retitle the section to reflect that it is a smoke test, not a benchmark.
• Add a benign control corpus, and report the false-positive rate alongside the block rate.
• Grade on a structured signal rather than refusal text: did the forbidden tool call happen, was the marker string emitted, did the secret reach the output. Say plainly why substring-matching the response is fragile.
• Report an interval, not a bare percentage, and state the corpus size and where it came from.
• Add two or three sentences on comparing defenses: if the intervals overlap, the corpus cannot rank them, and reporting the ranking anyway is the most common way these evaluations mislead.

Happy to write it and open the PR against this file only.

Disclosure, per the Use of AI section in CONTRIBUTING: I used an AI assistant to help structure and draft this issue. The measurements cited — the 66-payload corpus, the tier results, and the paired difference with its interval — are from my own work.