In short
A conventional penetration test examines code and infrastructure. It does not test how an AI system handles untrusted content that contains instructions, whether retrieval respects permission boundaries, or whether tool permissions are scoped tightly enough to limit damage from a manipulated request. AI systems require assessment of model behaviour in addition to standard application testing.
A different shape of exposure
Standard application security asks whether an attacker can reach data or functionality they should not. AI features introduce a variation on the question: whether an attacker can persuade a legitimate, correctly authenticated system to do it for them.
The ingredients are common to almost every enterprise AI deployment. The system reads broadly across internal data. It accepts natural-language input. It calls tools that do things. Individually these are ordinary. Together they create a path that conventional testing does not look for.
Prompt injection, stated plainly
Prompt injection occurs when content the system reads as data is treated as instruction. The content need not come from the user. A document uploaded by a third party, a web page the system retrieves, an email in a monitored mailbox. Any of these can carry text addressed to the model.
Because the system is authenticated as a legitimate user or service, the resulting action inherits that authority. A scanner will not find this. It is not a code defect; it is a consequence of how instruction-following systems work.
It cannot be eliminated, only contained
Any vendor claiming to have solved prompt injection should be treated with caution. The realistic objective is to make a successful injection low-impact, which is an architectural problem rather than a filtering one.
The controls that matter are unglamorous. Keep untrusted content structurally separate from instructions. Scope tool permissions to the minimum the task requires. Require human confirmation for consequential or irreversible actions. Validate outputs against expected structure. Enforce retrieval permissions per query against the source system. Log prompts, tool calls and outputs so an incident can be reconstructed.
Retrieval boundaries deserve specific attention
A frequent and serious defect is an index built with elevated privileges and queried on behalf of users with lower ones. The AI system then becomes an elegant mechanism for reading documents the user could not otherwise open.
Permissions must be evaluated at query time against the source system, not inherited from whatever account built the index. This is worth verifying explicitly in any AI deployment, including third-party products.
Third-party AI is still your exposure
Purchasing an AI product does not transfer responsibility for what it can read, what permissions it holds in your environment, or how its output is used. Assess bought AI features with the same seriousness as built ones, and ask specifically about data handling, retention and whether your content contributes to provider model training.
A practical starting point
Inventory where AI is already deployed, including features enabled inside products you already licence. For each, record what it can read, what tools it can invoke, what untrusted content it processes and what is logged. Most organisations find something in that inventory they were not aware of, and the exercise usually pays for itself before any testing begins.
Written by Mr. Rohit
Director and Chief Technology Officer, Acmez Technologies Pvt. Ltd.
This article reflects delivery experience on client engagements rather than vendor research. Where a claim cannot be substantiated, it is stated as an opinion or omitted. Last reviewed 30 July 2026.
About our leadership team
