Short answer

Data poisoning means manipulating the data an AI learns from or retrieves, so that it behaves badly or hides a backdoor that only a trigger activates. OWASP also warns about tampered models and weak provenance in the AI supply chain. You cannot inspect a model yourself, so choose reputable providers, avoid unknown model files and verify important answers.

Key takeaways

  • OWASP defines data poisoning as manipulation of pre-training, fine-tuning or embedding data to introduce vulnerabilities, backdoors or biases.
  • A poisoned model can carry a dormant backdoor that only a specific trigger activates, which makes it hard to find by testing.
  • OWASP warns that models shared through open repositories can carry malware, such as code hidden in malicious pickle files that runs when the model loads.
  • OWASP calls models 'binary black boxes' where static inspection offers little security assurance, and recommends integrity checks with signing and file hashes.
  • NIST's adversarial machine learning report and CISA's AI data security guidance both treat data integrity as a core security concern across the AI lifecycle.

What data poisoning is

Every AI system depends on data: the text, images and code it learned from, the documents it retrieves, and the files it loads. If someone tampers with that data, the system can be pushed into wrong, biased or harmful behaviour. OWASP’s LLM04:2025 entry defines data poisoning as manipulation of pre-training, fine-tuning or embedding data to introduce vulnerabilities, backdoors or biases, and classes it as an integrity attack.

The data can be poisoned at several stages. OWASP notes the risk is highest with external data sources, which may hold unverified or malicious content. Its scenarios include a competitor or malicious actor supplying falsified documents, and weak filtering that lets prompt injection insert misleading data.

The sleeper agent problem

The most worrying form is the backdoor. OWASP describes a poisoned model that works normally until a specific trigger activates hidden behaviour, which makes it hard to test for and creates what the page calls a “sleeper agent” risk. A planted trigger could, in its words, enable authentication bypass, data exfiltration or hidden command execution. This is a concern mainly for organisations building or customising models, but its effects reach customers through the products they ship.

The supply chain: where models come from

Few teams train a model from scratch. They download pre-trained models, adapters and datasets, often from public repositories. OWASP’s LLM03:2025 entry on supply chain lists the weak points: models are “binary black boxes” where static inspection offers little security assurance, model cards do not verify origin, attackers can compromise a supplier’s account or impersonate one, and a malicious adapter can undermine a base model. On-device models can be reverse engineered and re-packaged with tampered models, and unclear terms can lead to sensitive data being used for training.

The LLM04 page adds a plain warning about files: models shared through open repositories can carry malware, such as code hidden in malicious pickle files that runs when the model loads. For a hobbyist, that means downloading a model file is closer to running a program than to opening a document.

What the standards bodies say

NIST’s adversarial machine learning report (AI 100-2 E2025, published in March 2025) provides a taxonomy and terminology of attacks, and describes methods for mitigating and managing their consequences across the AI lifecycle. It exists so that standards and practice guides share a common vocabulary.

CISA’s AI data security guidance says data security is critical to the accuracy, integrity and trustworthiness of AI outcomes, and outlines risks that arise across all phases of the AI lifecycle, from development and testing to deployment and operation. The UK NCSC’s guidelines for secure AI system development, whose secure development stage covers supply chain security, align with practices from the NCSC, NIST and CISA.

For builders, OWASP’s mitigations are concrete: track data origins and transformations with tools such as an ML bill of materials, vet data vendors, use data version control to detect manipulation, test with red teaming, and monitor model behaviour. For model integrity it recommends signing and file hashes, and evaluating AI bills of materials, which it calls an emerging area.

What this means for you

You cannot audit a model’s training data, and reputable providers carry most of this burden. Your part is smaller but real. Install AI tools from official sources, and be suspicious of “free” or “unlocked” versions, which fit the pattern we describe in fake AI apps and malicious browser extensions. Do not load model files from strangers. Check answers that carry real stakes, a habit also covered in AI hallucinations and fake citations.

Connected tools widen the supply chain further, as discussed in MCP and AI tool connections. If you suspect you installed something malicious, disconnect it, change passwords from a clean device and follow the data breach checklist.

What you can do

  1. 01Download models and AI tools only from the publisher's official site or a repository you can verify, and avoid anonymous re-uploads.
  2. 02Do not load model files from strangers on a computer that holds work files or saved logins.
  3. 03Check an AI answer that matters, such as medical, legal or money advice, against a trusted primary source.
  4. 04Prefer providers that publish how they source and test training data and how to report problems.
  5. 05If you or your workplace fine-tune or feed an AI your own documents, restrict who can add to that data and keep a record of where it came from.
  6. 06Treat unexpected, oddly specific or repeated wrong answers as a reason to stop relying on that tool and report it.

Related reading: AI voice clone scams and the data breach tracker.

Frequently asked questions

What is data poisoning in simple terms?

It is tampering with the information an AI learns from or looks up, so its behaviour changes. OWASP defines it as manipulation of pre-training, fine-tuning or embedding data to introduce vulnerabilities, backdoors or biases.

What is a backdoor in an AI model?

OWASP describes a poisoned model that behaves normally until a specific trigger appears. That makes it hard to catch in testing, and the page calls it a sleeper agent risk.

Can an ordinary user spot a poisoned model?

Rarely. OWASP says models are binary black boxes where static inspection gives little assurance. That is why provenance, signing and trusted sources matter more than trying to inspect a model yourself.

Does this affect people who only use a chatbot?

Indirectly. You rely on the provider's data and testing, so you cannot check them yourself. Verify important answers and use established providers. The higher risk is for people who download model files or run unofficial AI tools.

Sources

  1. LLM04:2025 Data and Model Poisoning, OWASP GenAI Security Project
  2. LLM03:2025 Supply Chain, OWASP GenAI Security Project
  3. Adversarial Machine Learning: A Taxonomy and Terminology of Attacks and Mitigations (NIST AI 100-2 E2025), NIST, 2025-03-24
  4. AI Data Security: Best Practices for Securing Data Used to Train and Operate AI Systems, CISA
  5. Guidelines for secure AI system development, UK National Cyber Security Centre (NCSC), 2023-11-27

This article reflects public information as of 2026-11-06. AI tools and rules change quickly; we update articles when new facts are published. Spot an error? See our editorial and corrections policy.