Skip to main content

Introduction to AI Security

AI Security is the practice of assessing the security posture of machine learning systems, large language models (LLMs), and the applications and infrastructure built around them. As AI features move from research demos into production products, the attack surface has shifted. The model itself is only one component, and most real failures live in the seams between the model, the data it retrieves, the tools it can call, and the humans who trust its output.

Traditional application security still applies: authentication, authorization, injection, secrets handling, and supply chain. AI systems simply add attack classes that have no clean equivalent in classic web or mobile testing. A prompt is code-like input parsed by a non-deterministic interpreter. Training and retrieval data is executable influence. An agent with tool access is a confused deputy by design.

What this category covers​

  • LLM application security: prompt injection (direct and indirect), system prompt leakage, jailbreaks, output handling, and insecure output rendering.
  • RAG and vector stores: corpus poisoning, cross-tenant retrieval, embedding-layer weaknesses, and data leakage through similarity search.
  • Agentic systems: tool and function calling abuse, excessive agency, confused-deputy chains, MCP servers, and exfiltration through side channels.
  • Model and data supply chain: model provenance, model file formats, unsafe deserialization during model loading, and poisoned or backdoored weights.
  • AI infrastructure: inference endpoints, hosting layers, exposed model registries, notebook servers, and API keys leaking through AI tooling.
  • Governance and testing: threat modeling for AI features, evaluating guardrails, and reporting AI findings with real impact.

How to approach an AI target​

  1. Map the trust boundaries: where does untrusted text enter, and what can the model reach on the other side?
  2. Identify the blast radius: read-only chat, retrieval over private data, or tool execution against internal systems?
  3. Treat model output as untrusted input to every downstream consumer (browser, shell, SQL, another agent).
  4. Prove impact with a reproducible chain. A jailbreak that produces rude text is not a vulnerability. A jailbreak that reads another tenant's documents is.
warning

Model behaviour is probabilistic. Any finding in this category should be reproducible enough that a triager or client can observe the same impact more than once, ideally with a deterministic trigger.