Short answer

AI coding assistants can produce code that looks correct but is not secure, and GitHub's own documentation says users are responsible for reviewing and validating suggestions. OWASP warns that model output must never reach a shell or interpreter unchecked, and that models can invent package names an attacker may register. Review, test and sandbox AI-written code like a stranger's.

Key takeaways

  • GitHub's responsible use documentation says Copilot suggestions may look valid but not be secure, and that users should review and test generated code, especially in security-sensitive work.
  • OWASP says to treat the model as an untrusted user and never pass raw model output to a shell, eval or similar interpreter.
  • OWASP also warns that models can hallucinate package names, which attackers may then register with malicious code.
  • An assistant that reads untrusted content, such as a pasted issue or web page, can be steered by hidden instructions, and the UK NCSC says prompt injection may never be totally mitigated.
  • The less access an assistant has (read-only, one folder, no production keys), the less damage a bad suggestion or an injected instruction can do.

Why AI-written code needs a second look

AI coding assistants now suggest lines, write whole functions and, in agent mode, run commands and edit many files. That speeds up work for professionals and lets beginners build things they could not before. It also means code is arriving in your project that nobody carefully wrote or read.

The tools’ own documentation is direct about this. GitHub’s responsible use page for Copilot code completion says suggestions may appear valid without being semantically or syntactically correct, and that generated code “may not always be secure”. It states that users are responsible for reviewing and validating suggestions, and advises always reviewing and testing generated code, especially in security-sensitive applications. It also says users assume the risks of generated code, including security vulnerabilities and bugs.

Risk one: plausible but unsafe code

Insecure code rarely looks broken. A login function can run fine and still store passwords badly. A database query can work and still be open to injection. GitHub’s page names hard-coded passwords and SQL injection as the kind of problems normal secure coding and code review practices should catch. The fix is boring and effective: tests, linters, and a human who reads the diff.

Risk two: output that reaches a system unchecked

OWASP’s LLM05:2025 entry on improper output handling describes what happens when a model’s output is passed to other components without validation. Its examples include output sent to a shell or to functions such as exec or eval, which can lead to remote code execution, and generated SQL run without parameterisation. OWASP’s advice is to treat the model as an untrusted user and validate what it produces before it reaches anything that acts. For an individual, that means not pasting a long command into your terminal because an assistant produced it. Read it first.

Risk three: invented packages

OWASP also notes that models can hallucinate package names, which attackers may register with malicious code. The danger is a quiet one. You ask for a library, the assistant names one that sounds right, and a single install command runs someone else’s code on your machine. Search the package registry yourself, check how long it has existed and who publishes it, and prefer well-known options. This is the same suspicion you would apply to a fake AI app or malicious browser extension.

Risk four: an agent that can be steered

Agent-style assistants read files, issues, web pages and documentation. OWASP’s prompt injection entry explains that when a model accepts input from external sources, hidden instructions inside can change its behaviour. The UK NCSC says prompt injection attacks “may never be totally mitigated” and recommends reducing impact with deterministic controls, for example that a model reading outside content should not hold privileged tools. Our prompt injection explainer covers the basics.

OWASP’s excessive agency entry names the usual causes of damage: too much functionality, too many permissions and too much autonomy. It recommends avoiding open-ended tools such as arbitrary shell execution where a narrower tool will do, using least-privilege access, and requiring human approval for high-impact actions. See AI agent permissions and risks and MCP and AI tool connection risks for the connection side.

Risk five: secrets and private data

Anything you paste into an assistant may be stored by the provider under its terms. API keys, database passwords and customer records do not belong in a prompt. Keep them in environment files the assistant is told not to read, and rotate any key you have exposed. If a secret does leak, treat it as a breach and follow our data breach checklist.

A safe working routine

Work in a sandbox or a separate project without production access. Give the assistant the narrowest permissions that let it do the job. Keep command approval on. Verify every new dependency. Run tests and read the diff before you commit. None of this removes risk, but it moves a mistake from a breach to an annoying afternoon. Scammers also target developers with fake coding tools and “helpful” browser add-ons, so install assistants only from the vendor’s own site.

What you can do

  1. 01Read every AI-written change before accepting it, and run your normal tests and code review as if a stranger had written it.
  2. 02Check that each package the assistant suggests exists, has a real history and a publisher you recognise, before installing it.
  3. 03Never paste passwords, API keys or customer data into a coding assistant, and keep secrets out of files the assistant can read.
  4. 04Run generated scripts and commands in a sandbox, container or throwaway environment first, with no access to production systems.
  5. 05Keep command approval switched on for terminal, file deletion and network actions, and do not auto-approve new tools.
  6. 06Be wary of asking an assistant to act on untrusted text such as a stranger's issue, README or web page.

Related reading: AI voice clone scams and the data breach tracker.

Frequently asked questions

Is code written by an AI assistant less secure than human code?

The sources reviewed here do not give a single rate, so no honest blanket answer exists. GitHub's documentation says suggestions may look correct without being secure and that users should review and test them. Treat the output as unreviewed code from a source you cannot audit.

What is a hallucinated package and why does it matter?

A model sometimes names a software package that does not exist. OWASP notes that attackers may register such names with malicious code, so installing a suggested package without checking it can pull in malware. Look the package up and check its publisher and history first.

Can an AI coding agent be tricked by a web page or an issue?

Yes. OWASP describes indirect prompt injection, where hidden instructions in a website or file change a model's behaviour. If the agent also holds tools such as a shell or file access, an injected instruction may be carried out with those tools.

What is the safest way for a beginner to use a coding assistant?

Use it in a throwaway project or sandbox, keep secrets out of reach, read what it writes, and keep approvals on for commands. Do not run code you do not understand on a computer that holds work files or saved logins.

Sources

  1. Responsible use of GitHub Copilot code completion, GitHub Docs
  2. LLM05:2025 Improper Output Handling, OWASP GenAI Security Project
  3. LLM06:2025 Excessive Agency, OWASP GenAI Security Project
  4. Prompt injection is not SQL injection (it may be worse), UK National Cyber Security Centre (NCSC), 2025-12-08

This article reflects public information as of 2026-11-06. AI tools and rules change quickly; we update articles when new facts are published. Spot an error? See our editorial and corrections policy.