visa-vulnerability-agentic-harness

Visa Vulnerability Agentic Harness — Agentic Vulnerability Discovery, Remediation, and Validation

VVAH is Visa's open-source harness for autonomous vulnerability discovery, remediation, and validation using large language models, built on learnings from Project Glasswing (Anthropic's initiative for AI-assisted vulnerability research).

VVAH supports a four-phase pipeline: an S0 static seed followed by detection and reporting (S1–S9), with optional remediation and validation (S10–S11). The shipped default.yaml and full.yaml profiles derive that seed from the model; taint.yaml uses rules mode, which needs operator-supplied source/sink YAML. The four phases:

Phase 1 — Discovery & Modeling (S1–S3): map the attack surface and build a threat-aware plan.
Phase 2 — Deep Dive & Verification (S4–S6): run multi-lens analysis and adversarial verification to assess likely exploitability.
Phase 3 — Synthesis & Reporting (S7–S9): deduplicate, chain, and emit structured findings (Markdown + SARIF).
Phase 4 — Remediation & Validation (S10–S11, optional): propose candidate fixes and adversarially validate them before adoption. Disabled in the shipped default profile, which stops after Phase 3 (S9). See Run your first scan for how to enable these stages.

Three design choices drive finding quality: threat modeling before analysis focuses the attack surface; multi-agent deterministic voting reduces false positives; and structured triage artifacts compress the lifecycle from AI-discovered weakness to actionable finding. The bottleneck in AI-assisted vulnerability management is triage speed, not discovery. VVAH is designed around that constraint. The primary effectiveness metric is Mean Time to Adapt (MTTA): elapsed time from AI-discovered exploitability to a validated fix in production.

Multi-model by design. Every model-driven role — S0 through S11 — can be pointed at its own model and provider:

Anthropic Claude — native route
OpenAI-compatible — OpenAI models and any OpenAI-compatible gateway
Open-weight — served over Chat Completions-compatible endpoints

No single provider is a hard dependency. Detection and remediation/validation draw on the same set; see docs/models.md for the per-role matrix.

For setup, see docs/SETUP_GUIDE.md. GitHub Issues are open for bug, documentation, setup, and feature-request reports — see Reporting issues. This repository is not currently accepting external code contributions; see CONTRIBUTING.md for details.

Authorized use only. Run scans only against code you own or have explicit permission to test. Findings and fixes are LLM-generated triage candidates that require human review — see Limitations.

Data egress warning. Any role routed to via: cli, via: sdk, via: openai, or via: deepagents sends prompt data to that model provider endpoint (Anthropic/OpenAI or your configured gateway). Use only approved endpoints and scan targets you are authorized to process.

Docs: docs/ — full documentation index · SETUP_GUIDE.md — install & configuration · USER_GUIDE.md — commands & options · models.md — model/backend selection · remediation.md · validation.md · Project Glasswing white paper — technical background.

What's new in 1.4

Exploit verification (S6) — VVAH can now prove a finding is exploitable by attacking it with real HTTP traffic, so a confirmed finding arrives with evidence attached instead of waiting on a human to reproduce it. That is a direct move on MTTA: confirming exploitability is usually the slowest manual step between an AI-discovered weakness and a validated fix. Additive and positive-only — it never drops a finding, and the static verifier keeps the verdict. Beta, API-only, localhost-only, and off unless EV_API_COLLECTION is set — see Exploit verification below for the full safety constraints.
vvaharness ev-replay — regression-test your fix with the exploit that found the bug. After you patch and redeploy, replay re-runs the confirmed attack against the target and tells you whether it still lands — a regression test derived from the attack itself, not from a guess about it. We are not aware of another open-source harness that closes that loop.
--stop-after ev — validate a collection and target with no model spend
Exploit-verification credentials — EV's own auth credentials are read only from EV_AUTH_* environment variables, never from a profile or a CLI flag, and a collection may name the auth scheme but never a value. Other secrets can still reach the target on the wire: a token saved inside the collection's own requests, the client-key passphrase EV_TARGET_CLIENT_KEY_PASSPHRASE, and any hardcoded credential the attacker agent finds in the repo and replays.

Features

From detection to a graded fix — the S0–S11 pipeline combines threat- modeled discovery, deep-dive analysis, adversarial verification, reporting, remediation, and fix validation.
Threat-aware analysis — threat modeling and multi-lens research focus review on the attack surface that matters.
Reachable-code analysis — AST/call-graph seeding focuses model review on relevant code paths rather than the whole repo.
Real data-flow evidence — interprocedural taint analysis complements model-based review across supported application languages.
Coverage backstop — unrecognised files can be added to catch-all review chunks instead of being silently excluded. The backstop is best-effort, not a guarantee; see docs/security.md for coverage guidance.
Proof, not just suspicion (Beta — API only) — optional exploit verification attacks a candidate finding over live HTTP and reports back whether it is really exploitable, and ev-replay re-runs that same exploit after you ship the fix. Off unless you arm it; see Exploit verification.
Actionable output — produce Markdown and SARIF 2.1.0 reports with CVSS, CWE, and per-run diagnostics.
Your models and gateway — use Anthropic or OpenAI-compatible models through cli, sdk, openai, or deepagents routes, configured per role.
Portfolio-scale operations — scan CSV-defined repositories with resumable state and monitor long-running work through stderr output or the optional --s6-progress-file artifact.

See docs/features.md for the full capability reference, including backend and stage details, specialist lenses, taint analysis, and limitations.

Quick start

Prerequisites

Python 3.11 or newer.
Permission to scan the target repository.
One Anthropic credential for the packaged default profile: ANTHROPIC_API_KEY, ANTHROPIC_AUTH_TOKEN, or ANTHROPIC_SDK_API_KEY. Claude Code CLI auth (run claude, then /login; or claude setup-token) is optional and needed only for a role you configure with via: cli; no shipped profile does.

Credential and profile details are in docs/SETUP_GUIDE.md and docs/models.md.

Install and configure

git clone https://github.com/visa/visa-vulnerability-agentic-harness VisaVulnerabilityAgenticHarness
cd VisaVulnerabilityAgenticHarness

macOS / Linux

python3 -m venv .venv
source .venv/bin/activate
pip install .

Windows PowerShell

python -m venv .venv
.\.venv\Scripts\Activate.ps1
pip install .

cp .env.example .env
$EDITOR .env                       # set an Anthropic credential
claude                              # optional: only for via: cli roles — /login in the REPL (or: claude setup-token)

pipx install . is an alternative. For platform-specific setup, TLS/proxy notes, editable installs, and profile selection, use docs/SETUP_GUIDE.md.

Check the installation

vvaharness --version
vvaharness setup                    # readiness check; no model spend
vvaharness doctor                   # live backend check; spends model tokens

For the full configuration reference, see docs/configuration.md and docs/models.md.

Run your first scan

Step 1 — Detection only, no code edits. Always start here:

vvaharness scan --repo /path/to/your/repo --stop-after s9

Step 2 — Review the Markdown report and SARIF output in /path/to/your/repo/security-scan/.

Step 3 (optional) — When ready, explicitly enable remediation in a scan. This can edit source files:

vvaharness scan --repo /path/to/your/repo --remediate

The packaged default sets step_remediate.enabled: false and step_validate.enabled: false. --remediate enables S10 only, not S11; run standalone validate afterward to grade the fixes. In-scan S11 requires step_validate.enabled: true in the effective config (no scan --validate flag). Other configs or a local overlay can change these defaults; keep --stop-after s9 to explicitly skip S10/S11 with any profile.

Alternatively, after a detection-only run, remediate and validate independently:

vvaharness remediate --repo /path/to/your/repo --mode report-only  # propose only
vvaharness remediate --repo /path/to/your/repo                     # applies fixes
vvaharness validate --repo /path/to/your/repo                      # validate fixes

For command flags and troubleshooting, see USER_GUIDE.md.

Use with an AI agent (Claude / Copilot / Gemini)

vvaharness setup --install-agents

This writes AGENTS.md (cross-tool) and .github/copilot-instructions.md (Copilot) unconditionally; when the claude CLI is detected it also writes CLAUDE.md + a Claude skill in ~/.claude/skills/ (Claude Code), and when the gemini CLI is detected it writes GEMINI.md (Gemini CLI). Existing files are left untouched. See docs/SKILLS.md for the analysis capabilities.

Pipeline

VVAH implements an S0 static seed plus an eleven-stage workflow. The common operator path is detection through S9. The shipped default profile disables S10 remediation and S11 validation; sdk and full still enable both, while taint also leaves them off. These stages remain available by explicit opt-in or standalone commands.

Stage group	Stages	Purpose
Static seed (optional)	S0	Source/sink callgraph seed for taint-first scanning
Discovery & Modeling	S1–S3	Attack surface mapping, threat modeling, hunting plan
Deep Dive & Verification	S4–S6	Multi-lens research, policy gates, adversarial verification
Synthesis, Chaining & Reporting	S7–S9	Deduplication, chain construction, SARIF emission
Remediation & Validation	S10–S11	Candidate fixes and adversarial fix validation

For stage-by-stage internals, see docs/architecture.md. For every command and flag, see docs/USER_GUIDE.md.

Skills

The pipeline combines stage prompts, language-specific lenses, specialist security lenses, remediation playbooks, and validation personas. See docs/SKILLS.md for the full capability inventory, and docs/features.md for how those capabilities are selected by configuration.

Exploit verification (S6) — Beta (API only)

What you get. Exploit verification pro

visa-vulnerability-agentic-harness

工具介绍