Free model endpoints are seductive.
Zero cost.
Zero setup.
Zero reason to trust them.
I don't trust demos.
I trust failure modes.
So I built a small evaluation harness.
It tests one thing: can a free model endpoint gate a pull request for secrets?
This is not a benchmark.
It's a repeatable experiment.
You can run it in an afternoon.
Disclosure: This article was prepared as part of MonkeyCode's product outreach.
I used MonkeyCode's free model endpoint and the free server option for the test.
No quotas.
No hardware claims.
Just a harness and a rubric.
Why I stopped trusting free endpoints Free endpoints look great in a demo.
You paste a diff.
The model finds the secret.
Everyone claps.
Then you wire it into CI.
The JSON breaks.
The latency spikes.
The model misses a private key.
The demo didn't show that.
An evaluation will.
The experiment I designed a 40-minute test.
It answers one question: where does the free endpoint perform well, and where does it break?
The dataset is 30 synthetic diffs.
Fifteen contain real-looking secrets.
Fifteen are clean.
Each diff is small.
Each diff has one clear change.
The prompt is strict.
The model must return JSON.
No prose.
No apologies.
Just a verdict.
The harness is deliberately small.
It measures five things.
Accuracy.
JSON validity.
Latency.
Variance.
Failure modes.
The rubric Every answer gets one of five labels.
Label Meaning True positive Secret found, line and type correct False positive Clean diff flagged as secret False negative Secret missed Parse error Model returned prose or broken JSON Timeout No answer before the cutoff A good gate needs high recall.
A false negative ships a secret.
A false positive blocks a merge.
Both are expensive.
The harness makes them visible.
What the output looks like After the runs, the harness prints a scorecard.
Here's the shape.
Diff Run 1 Run 2 Run 3 Verdict diff_01 TP TP TP stable diff_02 FP FP TN unstable diff_03 parse parse TP broken diff_04 FN FN FN consistent miss This is a template, not a benchmark.
Fill it with your own numbers.
The pattern tells you more than the average.
A consistent miss is worse than a parse error.
A parse error is visible.
A miss is silent.
Where to look for strengths The harness will show you where the endpoint performs well.
Start with simple patterns.
AWS access keys.
GitHub tokens.
PEM blocks.
These have strong signals.
If the endpoint handles these, you have a useful gate.
Short diffs help.
One file.
One change.
No context to confuse it.
The free server option also matters.
You don't need a GPU.
You don't need a queue.
You send requests and wait.
For a small team, that's enough.
Where to expect breakage Watch for three failure modes.
Long diffs are the first.
The model can lose context.
It starts guessing.
It flags a base64 string as a secret.
Or it misses the secret in line
400.
JSON drift is the second.
The model wants to explain itself.
It returns prose.
It adds a period.
It wraps the JSON in a code block.
Your parser breaks.
Your CI fails.
The demo never showed that.
Variance is the third.
The same diff can get different answers.
Run one says true.
Run two says false.
Run three says true.
A single run is a coin flip.
You need agreement.
The decision table Use this table when you run the harness.
Observation Action Parse rate below 95% Add schema validation and one retry False negatives on PEM blocks Add a regex pre-filter before the model p95 latency over 10 seconds Move to an async queue, not blocking CI Same diff flips across runs Require two-of-three agreement False positives on base64 Add an allowlist or a wider context window This table is the real artifact.
It turns a vague "AI is unreliable" into a decision.
You can automate each row.
Turning the scorecard into a CI gate Once you have the scorecard, you can build a gate.
The gate runs only on changed files.
It calls the endpoint.
It parses the JSON.
It blocks the merge on a true positive.
It logs false positives for review.
It never blocks on a parse error.
It retries once.
This is the minimal version.
Add two-of-three agreement for unstable diffs.
Add a regex pre-filter for known patterns.
Add a human review queue for uncertain cases.
Limitations and who should not use this This harness is not a benchmark.
It uses synthetic diffs.
Real diffs are messier.
Real secrets are nested in larger changes.
Real teams have different tolerances.
Do not use this approach for compliance.
Do not use it as your only secret scanner.
Do not use it on a high-throughput CI without a queue.
The free endpoint is a helper, not a guarantee.
Who should use it?
Small teams.
Side projects.
Teams that want a second pair of eyes on a PR.
Teams that can tolerate a false positive now and then.
The takeaway Free model endpoints are not magic.
They are tools with failure modes.
The failure modes are predictable.
That means they are testable.
That means you can build a gate around them.
Run the harness.
Print the table.
Find the breakage.
Then decide if the free tier is worth it.
The scorecard will tell you.