When a company launches an AI assistant or agent, the security question is often answered with “we will run a penetration test”. That is necessary, but it is not enough. Penetration testing and AI red teaming look for different kinds of weaknesses. Here is how they differ and how to combine them.
Penetration testing: is the system built securely?
A penetration test looks for exploitable weaknesses in applications, APIs and infrastructure: broken authentication, access control flaws, injection, misconfigured cloud resources, outdated components. The system either has the flaw or it does not, and the fix is usually in code or configuration.
AI red teaming: can the AI be made to misbehave?
AI red teaming tests the behaviour of the model and of everything connected to it. Testers use crafted inputs, hidden instructions in documents, multi-turn manipulation and tool abuse to make the system leak data, ignore its rules or take actions it should not. Results are probabilistic: an attack might work one time in twenty, and that can still be a serious risk.
The differences at a glance
| Penetration testing | AI red teaming | |
|---|---|---|
| Main target | Code, APIs, infrastructure | Model behaviour, prompts, retrieval, agents and tools |
| Typical findings | Access control flaws, injection, misconfiguration | Prompt injection, data leakage, excessive agency, jailbreaks |
| Nature of results | Mostly deterministic | Often probabilistic |
| Reference | OWASP Top 10, OWASP ASVS | OWASP Top 10 for LLM applications |
| Typical fixes | Code and configuration changes | Permissions, guardrails, approval steps, retrieval controls, monitoring |
Why most AI products need both
An attacker does not care which layer is weak. A prompt injection can be the first step, and a missing access check in the API behind the agent can be the second. Testing only one layer leaves the chain open. The most effective approach is a single engagement that covers the AI behaviour and the application and infrastructure it runs on, with one threat model and one report.
When to run each one
- Before go-live: both, scoped together.
- After a model, prompt or tool change: AI red teaming, plus an automated regression suite in your CI pipeline.
- After an infrastructure or API change: a targeted penetration test.
- Every year: both, as part of your security programme and as evidence for the EU AI Act and NIS2.
Next step
Setexe combines AI red teaming and penetration testing in one engagement, run by engineers who also build AI systems. See our AI security services, our penetration testing services or talk to an expert.
