Local security audit for AI API relays and LLM proxies: detects prompt injection, model substitution, tool-call rewriting, SSE anomalies, error leakage, and Web
Local security audit for AI API relays and LLM proxies: detects prompt injection, model substitution, tool-call rewriting, SSE anomalies, error leakage, and Web
Local security audit for AI API relays and LLM proxies.
DSH Plugin
Your Agent Is Mine documents malicious API relays injecting payloads and exfiltrating credentials. Anthropic's September 10, 2026 report describes fraudulent Claude resellers swapping models and harvesting credentials through their client tooling. In his September 11 disclosure, co-author Chaofan Shou reports buying router data containing users' credentials.
API Relay Audit is an independent, local security audit tool for AI API relays and LLM proxies, informed by the paper. The current release checks observable relay behavior and generates a Markdown report covering:
LOW / MEDIUM / HIGH summaries;
inconclusive probes remain visible.See the output: example report (synthetic fixture) · Try it: run a local audit · Coverage and limits
The standalone script uses Python's standard library plus curl. Your API key
is sent only to the relay URL you choose.
AUDIT_SCRIPT_REF=v2.4.0
curl -fsSL "https://raw.githubusercontent.com/toby-bridges/api-relay-audit/${AUDIT_SCRIPT_REF}/audit.py" -o audit.py
python audit.py --key --url --output report.md
# Web3 / wallet users
python audit.py --key --url --profile web3 --output report.md
See a public-safe fixture report: sanitized audit report.
Use master as AUDIT_SCRIPT_REF only when intentionally testing unreleased changes.
If API Relay Audit helps you evaluate a relay before sending real traffic, star the repository to follow new detector coverage and release-tested updates.
inconclusive as clean; blocked probes and ambiguous responses stay visible in the report.| Query family | User intent | Profile / steps | Evidence boundary |
|---|---|---|---|
| API relay audit | Audit a third-party relay, mirror, gateway, LLM proxy, or resale API before trusting traffic. | general by default; full for every probe |
Produces a local report, not a safety certificate. |
| Prompt injection audit | Detect hidden prompt injection, prompt leakage, instruction override, and extraction behavior. | general; Steps 3-6 |
Records prompt evidence without publishing private prompts or secrets. |
| Model substitution signals | Collect model identity, stream, latency, and upstream channel signals. | general; Steps 5, 10, 13, 14 |
Self-ID, latency, and channel fingerprints are signals, not standalone proof of provider substitution. |
| Web3 relay audit | Check wallet-sensitive relay behavior before agent workflows touch signing or transactions. | web3 or full; Step 11 |
Profile-gated; general relay audits do not imply wallet safety. |
The canonical contract lives in docs/query-families.md. README headings, Pages cards, issue templates, and skill descriptions should preserve these boundaries instead of flattening them into one slogan.
API Relay Audit checks whether a relay modifies the request or response path between you and the model:
The project has two distribution modes:
audit.py: zero-dependency standalone script for quick local auditsapi_relay_audit/ plus scripts/: modular development version with testsRuntime profiles:
general: default AI API relay and LLM proxy checksweb3: wallet-safety probes for Web3 agent flowsfull: general plus Web3 checksThe repository is also an installable dsh-api-relay-audit bundle for
DeepSeek Harness Web and
community TUI surfaces that use the official @deepseek-ai/dsh-commands
registry. Pin an immutable commit or release tag:
DSH_PLUGIN_REF=v2.4.0
dsh plugin --profile web add "github:toby-bridges/api-relay-audit#${DSH_PLUGIN_REF}"
# dsh-cc-tui and other compatible profile-based clients
dsh plugin --profile cc-tui add "github:toby-bridges/api-relay-audit#${DSH_PLUGIN_REF}"
The command reuses the current DSH provider's baseURL, model, and credential
reference. The credential stays in DSH Credentials and is delivered to the
local audit process through an environment variable, never through command
arguments or the session log:
/relay-audit
/relay-audit --connectivity
/relay-audit --profile web3 --fast-context
/relay-audit --url --model --credential-ref
No arguments preserves the existing full-audit default and may consume
metered tokens. Use --connectivity for a lower-cost check. This distribution
does not add a new model baseline: the selected route must identify as Claude,
although the relay API itself may be Anthropic-compatible or OpenAI-compatible.
Independent wrappers without DSH profiles and the DSH command registry are not
compatible with this bundle. See agent distribution notes.
The exact v2.4.0 installation, runtime, and secret-scan results are recorded in
the DSH distribution verification.
The repository retains its existing OpenClaw and Hermes skill files for direct users and downstream compatibility. They are not current registry distribution targets; active distribution and release verification focus on the DeepSeek Harness plugin.
These files do not certify that a relay is safe. They help agents generate a local, reviewable Markdown report before trusting a relay path.
Natural-language self-identification is treated as a consistency signal, not upstream proof. A response saying it is Qwen, DeepSeek, GPT, or Claude can indicate a mismatch, but it does not by itself prove that a provider substituted the upstream model.
Stronger claims require corroborating evidence such as raw response JSON, request IDs, provider/model metadata, stream signatures, transparent-log hashes, and reproducible runs. Public submissions should use redacted report artifacts and never include API keys, raw headers, full response bodies, wallet material, private relay traffic, or user data.
With --profile web3 or --profile full, API Relay Audit adds wallet-oriented prompt injection probes inspired by signature-isolation risks:
These probes are model-agnostic, but they are intentionally profile-gated so general relay audits stay focused.
your machine
-> audit.py / scripts/audit.py
-> chosen relay endpoint
-> Markdown report + optional hash-only transparent log
-> optional: redacted evidence issue for maintainer review
Community evidence is shape-checked by GitHub Actions, but publication still requires maintainer review. Operators keep a separate response path, and sensitive vulnerabilities belong in the disclosure path described in SECURITY.md.
| Metric | Current value |
|---|---|
| Version | v2.4 |
| Audit steps | 14 |
| Risk matrix | 6D |
| pytest collected tests | 808 |
| CLI flags | 22 |
| Runtime profiles | general, web3, full |
An API relay or LLM proxy is a third-party service between you and an AI provider such as Anthropic or OpenAI. It forwards your requests upstream, but it can also inject hidden instructions, swap models, truncate context, rewrite tool output, or leak credentials in error responses.
API Relay Audit runs locally, so your API key is sent only to the relay URL you specify. The standalone version is a single Python file with zero Python package dependencies, which makes it easier to inspect before running.
Prompt injection means the relay may prepend or insert hidden instructions into your request. API Relay Audit compares expected and actual token usage, tries prompt-extraction probes, and records evidence when the relay appears to add or reveal hidden prompt content.
Model substitution means the relay claims to provide one model but may expose evidence signals for another model identity, route, or upstream channel. API Relay Audit checks non-Claude identity patterns, anchor phrases, stream model identity, latency variance, and channel evidence where available; those signals require corroboration before making provider-level claims.
Tool-call rewriting means the relay modifies package-install commands or tool-like output in the model response. API Relay Audit sends pinned package commands and comp
No open issues yet, or sync has not completed.