Chatbots, internal assistants and AI agents are now connected to customer data, document stores and business systems. That makes them part of your attack surface. Classic penetration tests rarely cover how a large language model (LLM) behaves under adversarial input, so LLM applications need their own security testing. This guide explains what to test, how a test is run and what you should get at the end.
Why LLM applications need their own security testing
A traditional web application does what its code says. An LLM application also does what its input persuades it to do. Instructions hidden in an email, a web page or a PDF can change the model’s behaviour, and the model can pass that behaviour on to the tools it is allowed to call. The result is a new class of vulnerabilities that scanners built for SQL injection or cross-site scripting do not see:
- Prompt injection: direct or indirect instructions that override the system prompt.
- Sensitive data disclosure: the assistant reveals personal data, secrets or documents the user should not see.
- Excessive agency: an agent sends emails, changes records or spends money because it was tricked into it.
- Insecure output handling: model output is executed or rendered without validation.
What to test: the LLM attack surface
Scope the test around the whole system, not only the model:
- System prompt and guardrails: can they be extracted or bypassed?
- Retrieval (RAG) pipeline: can a poisoned document steer answers or leak other users’ content? Are access rights enforced at retrieval time?
- Tools and agents: which actions can the model trigger, with which permissions, and is there human approval for risky ones?
- Output handling: is generated HTML, code, SQL or Markdown sanitised before it is rendered or executed?
- Data and logs: are prompts and responses with personal data stored, and for how long?
- Supply chain: model providers, plug-ins, open-source models and libraries.
- Cost and availability: can an attacker drive up token usage or exhaust rate limits?
How an LLM security test is run, in five steps
- Scope and threat model. We map users, data, tools and trust boundaries, and agree which outcomes would really hurt the business.
- Automated adversarial testing. Large sets of known attack prompts and jailbreak patterns are run against the application to find weak spots quickly and repeatably.
- Manual red teaming. Experienced testers chain techniques together, as a real attacker would: indirect injection through documents, multi-turn manipulation, role-play and encoding tricks.
- Agent and tool abuse. We try to make the agent perform actions outside its mandate, escalate privileges or exfiltrate data through its tools.
- Report, fix and retest. Every finding comes with evidence, impact, a fix and a retest once it is fixed.
What a good report contains
- A short executive summary with business impact, not only technical scores.
- Findings mapped to the OWASP Top 10 for LLM applications, with reproducible prompts and responses.
- Concrete fixes: input and output controls, least-privilege tool permissions, human approval for sensitive actions, retrieval-time access checks and monitoring.
- A retest section that shows what was closed.
How testing supports the EU AI Act and NIS2
The EU AI Act asks providers of high-risk AI systems for a risk management system (Article 9) and for an appropriate level of accuracy, robustness and cybersecurity (Article 15), including resilience against attempts to manipulate inputs. Documented adversarial testing is direct evidence for both. Read our overview of the current EU AI Act timeline to see when each obligation applies.
If your company is in scope of NIS2, AI systems are part of the network and information systems you must protect. Their risks belong in your risk analysis, and incidents involving them follow the same notification rules. See NIS2 in Italy after October 2026.
How often to test
Test before go-live, after every significant change (a new model, new tools, new data sources) and at least once a year. Between tests, keep an automated regression suite of attack prompts in your CI pipeline, so a prompt or model update cannot silently reopen a fixed vulnerability.
Next step
Setexe tests LLM applications and AI agents with the same engineering team that builds them, so findings come with fixes that work in production. Learn more about our AI security testing and governance services or talk to an AI security expert.
