When Not to Use AI for Code

TL;DR: The honest answer to when not to use AI coding: security-critical code (auth, crypto, payment logic), a genuinely novel algorithm with no real precedent in training data, and code in a domain you don't understand well enough to catch a plausible mistake. Two separate studies back this up: AI-assisted developers write less secure code while feeling more confident about it, and code-generating models hallucinate non-existent packages often enough to create a new attack category.

When Should You Not Use AI for Code?

Most "AI coding limitations" advice hedges: use good judgment, review carefully, don't trust it blindly. That's true, and useless. Nobody sets out to trust it blindly. The more honest version of when not to use AI coding names three specific situations where the failure mode isn't "obviously wrong output you'll catch," it's "fluent, confident, plausible output that is wrong in a way you won't catch until it matters."

Those three situations are: code that's security-critical, code that implements a genuinely novel algorithm, and code in a domain you don't understand well enough to judge correctness yourself. None of them are about AI being bad at coding in general. It isn't. They're about a specific mismatch between what these models are good at (recombining patterns they've seen a lot of) and what the task actually needs (something new, something adversarial-proof, or something only a domain expert can verify).

The rest of this post backs each one with real evidence, not a hunch, and ends with a checklist for judging your own task and what to do when AI genuinely is the wrong tool.

Why Is AI-Generated Code a Security Risk?

Start with the closest thing to a controlled experiment on this question. Researchers at Stanford ran what their paper describes as "the first large-scale user study examining how users interact with an AI Code assistant to solve a variety of security related tasks across different programming languages" (Perry et al., "Do Users Write More Insecure Code with AI Assistants?", ACM CCS 2023, arXiv:2211.03622, accessed September 3, 2026). Participants split into two groups (one with access to an AI code assistant, one without) and worked on security-relevant tasks: string encryption, SQL query construction, file path handling, and similar.

The result is the part worth sitting with. Participants who had access to the assistant "wrote significantly less secure code than those without access." That alone would be a reason for caution. What makes it worse is the second finding: those same participants were "more likely to believe they wrote secure code than those without access to the AI assistant." The tool didn't just produce worse code. It made people more confident in code that was, on average, worse. The one group that did better with AI access were participants who "trusted the AI less and engaged more with the language and format of their prompts", which is a finding worth remembering for every other section of this post, not just this one.

The second piece of evidence is about the supply chain, not the logic. Code-generating models don't look up whether a package exists before recommending it. They predict a plausible-sounding name. A 2024 study across 16 popular code-generation models, both commercial and open-source, generated 576,000 code samples and found the average hallucination rate is "at least 5.2% for commercial models and 21.7% for open-source models," with "205,474 unique examples of hallucinated package names" across the dataset (Spracklen et al., "We Have a Package for You! A Comprehensive Analysis of Package Hallucinations by Code Generating LLMs," accepted to USENIX Security 2025, arXiv:2406.10279, accessed September 3, 2026).

This isn't only an accuracy problem. It's an attack vector with a name. "The term slopsquatting was coined by PSF Developer-in-Residence Seth Larson and popularized in a recent post by Ecosyste.ms creator Andrew Nesbitt" (Socket, "The Rise of Slopsquatting," accessed September 3, 2026). An attacker registers a real package under the exact name a model tends to hallucinate, waits, and collects installs from anyone who copy-pasted the AI's import statement without checking it first. A security review from the same AI, or from a code-review tool, won't reliably catch this either: GitHub's own documentation for Copilot's automated code review states plainly that it isn't guaranteed to catch every issue and that it excludes dependency-management files from what it scans, meaning the exact file where a hallucinated package name would land is outside its view by design.

None of this means every line of security-relevant code needs a security engineer standing over it. It means the confidence you feel reading AI-generated auth, crypto, or payment code is not evidence that it's correct, and a package name you didn't recognize before pasting it deserves thirty seconds of npm view <package> or pip index versions <package> before it deserves an install.

Why Can't AI Write a Genuinely Novel Algorithm?

This one isn't a benchmark statistic, and it follows from how these models are built. It's worth reasoning through plainly rather than dressing it up as a number nobody's actually measured for your specific problem. A code-generating model produces the most statistically likely continuation of the patterns in its training data. That's an enormous strength for anything with strong precedent: a REST endpoint, a binary search, a standard authentication flow, a common data transformation. There are thousands of public examples to draw from, and the model's job is close to interpolation between things it has effectively seen before.

A genuinely novel algorithm doesn't have that. If you're implementing something that hasn't been written before (a new approach to a scheduling problem, a custom compression scheme, a mathematical technique specific to your domain), there's no strong precedent for the model to draw on. What comes back tends to be a fluent recombination of the nearest familiar pattern, not the new logic your problem actually needs. The dangerous part is that it can look complete: correct syntax, plausible variable names, a docstring that describes what the code is supposed to do. Nothing about the output signals "I'm pattern-matching to something adjacent, not solving your actual problem."

Contrast this with what AI is good at inside the same coding session: chain-of-thought prompting can help a model reason more carefully through a known class of problem, and few-shot vs. zero-shot prompting can steer it toward a specific known pattern with examples. Neither technique manufactures a precedent that doesn't exist in training data. They make the model better at the kind of problem it was already suited to, which a genuinely novel algorithm, by definition, is not.

If you're not sure whether your problem is "novel" or just "unfamiliar to you," ask a narrower question: does a textbook, a well-known library, or a documented pattern already solve this class of problem? If yes, you're in AI's comfort zone, and a well-structured prompt (see chain-of-thought prompting for how to get a model to reason through the steps instead of guessing the answer) will genuinely help. If the honest answer is "no, I'm inventing this," the model is guessing too, just more fluently than you are.

Why Does AI Struggle with Unfamiliar-Domain Code?

The security and novelty cases are about the model. This one is about you. Code in a domain you don't understand well enough to judge is dangerous with AI for a reason that has nothing to do with the model's competence: you can't catch an error in logic you can't evaluate.

Every domain has correctness rules that live outside the code itself. A dosage calculation has a rounding convention that matters for patient safety. A tax calculation has a jurisdiction-specific rule that changes the right answer for two customers in different states. An embedded control system has a safety interlock that has to fail in a specific direction, not just "fail." A financial reconciliation has a rule about which side an off-by-one error should favor. None of that is visible in the code's syntax. All of it is invisible unless you already know to look for it. And if you already know to look for it, you're not in the unfamiliar-domain case this section is about.

An AI model trained mostly on general-purpose code has thinner coverage of any one narrow domain's specific rules than it does of common web-development patterns, simply because there's less of that narrow domain in public training data to draw from. That doesn't make the output look uncertain. It still reads as confident, well-commented, plausible code, the same failure mode as the security case, just with domain-specific correctness standing in for security. If you can't personally tell whether the rounding is right, the jurisdiction rule is applied correctly, or the interlock fails safe, neither reading the code nor asking the AI to "double-check it" gives you that ability. You need a person who already has it.

Where AI Code Generation Helps vs. Where It Doesn't

How Do You Know If Your Task Is Too Risky for AI?

Run your task against these four questions before you accept AI-generated code as anything more than a first draft:

1. Does this code touch a trust boundary? Authentication, authorization, cryptography, payment handling, anything that processes untrusted input from outside your system. If yes, treat the output as a draft that needs a security-qualified reviewer, not a finished answer.
2. Could I explain, from memory, why this approach is correct — not just that it runs? If the honest answer is "I'd have to ask the AI to explain it to me," you're not in a position to catch a subtle error in it.
3. Does a textbook, standard library, or documented pattern already solve this class of problem? If nobody has written this before, you're asking for a novel algorithm, and pattern-matching to something adjacent isn't the same as solving your actual problem.
4. Did the code introduce a package, library, or API I didn't already know existed? Verify it against the real registry before it goes anywhere near a dependency file:

# Node / npm — confirms the package actually exists and shows real version history
npm view <package-name>

# Python / pip — same check against the real PyPI index
pip index versions <package-name>

A "yes" on questions 1 or 3, or an honest "no" on question 2, means you're in one of the three categories this post covers. That's not a reason to abandon the task — it's a reason to change who verifies it and how.

What Should You Do Instead When AI Is the Wrong Tool?

Reach for an audited library before you reach for hand-rolled logic, whenever one exists. An established cryptography or authentication library has already survived years of adversarial review that your freshly generated function has not, AI-assisted or not. This is true whether a human or a model wrote the alternative — the point of using a maintained library isn't that it's untouched by AI, it's that it has already been attacked and patched by people whose job is finding exactly the class of bug you're worried about.

Where no library covers the case — a genuinely novel algorithm, or logic specific to a regulated domain — get a qualified human reviewer before it ships. Not a second AI pass: a person whose judgment you'd trust to review the same code if a colleague had written it by hand. Write the tests that define correct behavior before you accept any implementation, so "it runs" and "it's correct" stay separate questions. And treat every unfamiliar package name the same way you'd treat an unfamiliar dependency a colleague added to a pull request: look it up before it ships, not after.

None of this is an argument against AI-assisted coding generally. It's an argument for knowing which task you're actually doing. Boilerplate, well-documented patterns, and refactoring with tests to check your work are a different job than security-critical logic, a genuinely new algorithm, or code in a domain you can't personally verify — and the honest answer for that second group, more often than the industry likes to admit, is don't.