#5330·casdoor

[feature] support penetration test for LLM agents

Author: hsluoyzCreated Mar 26, 2026Updated Sep 14, 2026
Labelsenhancement

As terminal AI agents evolve from conversational tools into super-assistants with autonomous execution capabilities, the security paradigm has undergone a fundamental shift. GUI Agents not only generate content but also autonomously plan, decide, and execute across systems. While this high degree of autonomy improves efficiency, it also concentrates risk — once out of control, it can directly lead to data breaches, financial losses, or even physical system damage.

Traditional security measures (such as permission isolation, sandboxing, identity authentication, and secure communication protocols) can provide baseline protection but struggle to address AI-native threats including prompt injection, context pollution, multimodal adversarial attacks, long-term memory poisoning, and privacy exfiltration induced by compliant applications during autonomous execution. The essence of these attacks is context confusion and context pollution, rooted in a core vulnerability of AI Agents: ambiguous context boundaries. The model cannot reliably distinguish, verify, or isolate instructions and data of different trust levels — instructions and external content are flattened and mixed within the context window, breaking the strict boundary between code and data that exists in traditional programs.

To this end, this research targets the systemic security risks arising from ambiguous context boundaries in autonomous AI agents, focusing on high-risk attacks such as prompt injection, covert injection through multimodal inputs and long-term memory, induced operations, and privacy theft. The research aims to deeply integrate traditional security mechanisms with AI-native dynamic defense capabilities, constructing a unified defense guardrail that is endogenous to the agent's reasoning and execution pipeline. By systematically modeling attack paths, the approach enhances the agent's structured perception of context, trust-level assessment, and intent reasoning capabilities, enabling it to identify and isolate potentially malicious instructions and data in real time. Combined with traditional security measures such as permission control, sandbox isolation, and identity verification, this forms a lightweight, low-intervention, automated collaborative defense system that builds stronger security and privacy protection boundaries while preserving the agent's highly autonomous user experience.

The constructed context attack sample library or attack path taxonomy must cover ≥95% of known typical attack patterns (such as prompt injection, induced interaction, privacy theft, memory poisoning, etc.) to ensure the completeness of the knowledge base.

At least one defense solution must be provided.