How it works
A model receives developer instructions and external content in one undifferentiated token stream, with no structural marker separating authority from data. Content that changes the model's behaviour is therefore indistinguishable from instruction, and any capability the model holds becomes reachable by whoever controls that content.
Why this is not a filtering problem
Prompt injection is usually framed as detecting malicious text, and that framing is why the proposed fixes keep failing. An instruction is not a recognisable property of text — it is any text that changes what the model does next, and that set is as large as language itself.
A classifier tuned on imperative phrasing loses to a question. One tuned on English loses to another language, to an encoding, or to text that establishes a premise the model then acts on without ever being told to.
The root cause is architectural. The model receives one sequence of tokens. Developer instructions and external content arrive in the same channel with no structural distinction, so asking the model to honour a boundary is asking it to reconstruct information the interface never carried. It usually succeeds, which is precisely what makes it dangerous — the defence appears to work until an input happens to win.
Indirect injection is the realistic case
Direct injection, where a user types something adversarial into their own session, is largely a self-inflicted problem: the user manipulates a model acting on their own behalf with their own permissions.
Indirect injection is the one that matters. The payload is placed in content the model will later read on someone else's behalf — a web page it will summarise, a document in a shared drive, an email in an inbox, a code comment, a calendar invitation, a support ticket. The victim never sees the instruction and takes no action beyond asking for something ordinary.
This changes the threat model completely. Any content the model may ingest becomes an input channel from whoever authored it, which means a retrieval-augmented system inherits the trust boundary of every document in its corpus, and an agent reading email inherits it from every sender.
Agency is what converts a text problem into a security problem
A model that produces the wrong sentence is a quality issue. A model that produces the wrong sentence and can send mail, query a database, move money or modify infrastructure is an incident.
That distinction is the whole mitigation strategy. Injection is dangerous strictly in proportion to the capability behind the model, and capability is something engineers control precisely — unlike model behaviour, which they influence probabilistically.
Excessive agency is the term for granting more capability than the task requires: an assistant given write access because read-only was inconvenient to scope, a connector with tenant-wide reach because per-user tokens were more work, an agent permitted to chain tools without any step requiring confirmation. Each decision is individually reasonable and collectively determines the blast radius of a successful injection.
Constraining capability in practice
The controls that hold are ordinary authorisation engineering, applied on the assumption the model is untrusted.
- Narrow tool contracts. A tool executing arbitrary SQL is compromised the moment the model is; one that fetches an order by identifier, scoped to the authenticated user, is not. No argument the model can supply should produce an action the user could not perform themselves.
- Authority from the session. Whose data the model may touch is established by authentication, never by anything in the prompt. Passing an account identifier through the context turns injection into privilege escalation.
- Split reading from acting. One context ingests hostile content and emits a structured, validated result; a second acts on that result and never sees the original text. The attacker controls what the first reads but not the schema the second accepts.
- Confirmation by consequence. Irreversible or outward-facing actions require human approval regardless of how confident the model appears — a successfully injected model is often extremely confident.
- Schema-validated tool calls. If a call must be one of a known set with typed arguments in permitted ranges, much injected instruction becomes unrepresentable.
- Policy enforced outside the model, with an audit trail recording which action was taken, on whose authority, and what content was in context when it happened.
What to test
Testing against known payloads measures resistance to yesterday's phrasing. More useful questions, asked per tool rather than per prompt:
What is the worst outcome if the model invokes this tool with attacker-chosen arguments? If that answer is unacceptable, the tool is too broad, and no filtering in front of it changes that.
Can any content in context cause the system to act as a different user? If so, that is the finding; the injection technique is incidental.
Which actions are irreversible, and which of those can fire without human confirmation?
If the input classifier were removed entirely, what would change? If the answer is "everything", the classifier is load-bearing and the architecture is wrong.
Also known as
What defenders may observe
- Model output conflicts with the stated task
- Tool calls that do not correspond to the user's request
- Sensitive system, connector or other-tenant data appearing in output
- An agent invoking unrelated or high-impact tools
- Actions occurring with no matching approval record
- One connector providing lateral reach into another
Prevention and mitigation
- Treat every model input and output as untrusted
- Give tools narrow contracts — no argument the model supplies should exceed the user's own permissions
- Derive authority from the authenticated session, never from prompt content
- Separate the context that reads hostile content from the one that acts on it
- Require human approval by consequence, not by model confidence
- Validate tool calls against a schema and enforce policy outside the model with a complete audit trail
Initial response priorities
- Disable unsafe model actions or affected data connectors.
- Preserve prompts, retrieved content, tool calls and model outputs.
- Revalidate trust boundaries, permissions and human approvals before restoration.
Use your approved incident-response plan and adapt decisions to your environment. This is educational guidance, not a substitute for live incident leadership.