How it works
An attacker influences training, tuning or retrieval data before inference so behaviour is altered or a hidden trigger is embedded; or queries the deployed model systematically to reproduce its behaviour, extract memorised content, or recover proprietary capability.
Two attacks on one asset
A model is both a program and a dataset, and it can be attacked as either. Poisoning corrupts what it learned; extraction steals what it knows. They are grouped here because both target the model as an asset rather than the application around it, and because both are largely invisible to application-layer defences.
Neither produces an error. A poisoned model performs normally on everything except the inputs the attacker cares about. An extraction campaign consists entirely of valid, authorised, correctly-billed queries. Nothing in a conventional security stack is looking at either.
Poisoning and the trigger problem
Poisoning influences data before or during training. The general form degrades quality, which is noisy and self-limiting. The dangerous form is targeted: the model behaves correctly everywhere except on a narrow class of input, where it does what the attacker chose.
A backdoor trigger is the sharpest version — a specific phrase, token sequence or pattern that reliably produces attacker-selected behaviour, while aggregate evaluation metrics remain excellent because the trigger appears nowhere in the test set. Standard evaluation cannot find this by construction: you cannot test for a trigger you do not know exists.
The realistic entry points are rarely the base training corpus, which is expensive to influence. They are the parts of the pipeline that accept contributions continuously — fine-tuning datasets assembled from user feedback, retrieval corpora that ingest documents automatically, and third-party model artifacts downloaded from public registries.
That last one deserves emphasis. Pulling weights from a public repository is functionally identical to executing a binary from an untrusted source, and it is routinely done with less scrutiny than a dependency upgrade would receive.
Extraction
Extraction abuses legitimate access to reproduce something valuable. The variants differ in what is being taken.
Behavioural cloning queries the model systematically and trains a substitute on the responses. The copy need not match exactly to destroy the commercial value of the original, and this is entirely achievable within ordinary API terms.
Memorisation attacks target the training data rather than the model, probing for content the model reproduced verbatim. Where that corpus contained personal data, credentials or licensed material, the exposure is a data breach expressed through model output.
Boundary mapping probes the edges of a classifier to learn where its decisions change, which is directly useful for evading a security model — a spam, fraud or abuse classifier whose boundary is known is a classifier that can be walked around.
The detection signal is query distribution rather than query volume. Legitimate use clusters around real tasks; extraction sweeps systematically, probes rare regions, and shows unusual coverage of the input space. Rate limits alone will not catch a patient attacker staying under them.
Controls
Both halves come down to provenance on the way in and restraint on the way out.
- Know where every training and retrieval input came from, and restrict who can add to it. Treat automatic ingestion as an untrusted input channel, because it is one.
- Sign artifacts and verify at load. An unsigned model pulled from a registry is unverified code.
- Keep evaluation data the pipeline cannot see, and review per-slice rather than in aggregate — targeted poisoning is invisible in an average.
- Return the minimum useful output. Confidence values, token probabilities and verbose explanations all accelerate extraction substantially and are rarely necessary.
- Monitor query patterns for systematic coverage, not just for volume.
- Guard artifacts like credentials. Weights readable by any principal in the environment are weights that will eventually leave it.
Also known as
What defenders may observe
- Unexpected behaviour tied to narrow, specific inputs
- Dataset lineage, checksums or artifact signatures change
- Evaluation quality diverges for targeted cases while aggregate metrics hold
- High-volume systematic querying that maps decision boundaries
- Requests probing rare behaviours or reconstructing training examples
- Model artifacts read by unusual principals
Prevention and mitigation
- Establish dataset provenance and restrict who can contribute to it
- Sign model and data artifacts, and verify signatures at load
- Hold out evaluation sets the pipeline cannot see, and review anomalies per-slice
- Rate-limit and monitor query patterns, not just query volume
- Minimise output verbosity — confidence scores and logits accelerate extraction
- Control access to artifacts as strictly as to production credentials
Initial response priorities
- Disable unsafe model actions or affected data connectors.
- Preserve prompts, retrieved content, tool calls and model outputs.
- Revalidate trust boundaries, permissions and human approvals before restoration.
Use your approved incident-response plan and adapt decisions to your environment. This is educational guidance, not a substitute for live incident leadership.