What a code model actually learns

A code model is trained to predict the next token from everything before it. The training data is mostly public code: GitHub repositories, package source, blog tutorials and Stack Overflow answers. Nothing in that process labels code as secure or insecure. The model learns what tends to come next.

That works well for most programming. The usual way to sort a list is also a correct way to sort a list. Security is different. Tutorials skip input validation to keep the example short. A 2012 answer with string-built SQL still has thousands of upvotes. Demo projects with password = "admin" vastly outnumber hardened production services. So the most common pattern in the data is often the unsafe one, and the model ranks it highest.

# get a user by id from Postgres HOW OFTEN EACH PATTERN SHOWS UP IN PUBLIC CODE (ILLUSTRATIVE) f"... WHERE id = {id}" common, unsafe "..." % id less common, unsafe execute(sql, (id,)) safe, but not the most common The model picks what it saw most. Being correct and being frequent are different things.
Illustrative, not measured: the ranking idea, not real frequencies.

What the studies found

This isn't a guess. Several independent studies have measured it, with different setups and similar conclusions:

  • Pearce et al., "Asleep at the Keyboard?" (IEEE S&P 2022). The researchers prompted GitHub Copilot with 89 security-relevant scenarios and got 1,689 programs back. About 40% of them contained a vulnerability from MITRE's CWE Top 25.
  • Perry et al., "Do Users Write More Insecure Code with AI Assistants?" (ACM CCS 2023). Developers using an AI assistant wrote less secure code than the control group, and were more likely to believe their code was secure. That second finding matters most.
  • Fischer et al., "Stack Overflow Considered Harmful?" (IEEE S&P 2017). Before LLMs existed, security-related snippets copied from Stack Overflow showed up in about 15% of 1.3 million Android apps, and nearly all of those apps included at least one insecure snippet. That's the same public code models now learn from.
  • Veracode, GenAI Code Security Report (2025). Across more than 100 models, about 45% of generated code samples failed security tests. Newer and bigger models weren't meaningfully better at security, even though they were better at producing code that runs.

The numbers differ because the methods differ. The direction is the same every time.

The mistakes are predictable

The good news is that the failures aren't random. They cluster around the same patterns that are common in tutorials. Here's a small Flask endpoint of the kind an assistant writes in seconds when you ask for "a login route":

app.pyPython
app.config["SECRET_KEY"] = "dev-secret"          # 1. hardcoded, guessable

@app.post("/login")
def login():
    u = request.form["username"]
    p = request.form["password"]
    row = db.execute(
        f"SELECT * FROM users WHERE username = '{u}'"   # 2. SQL injection
    ).fetchone()
    if row and row["password"] == hashlib.md5(p.encode()).hexdigest():  # 3. MD5, no salt
        session["user"] = u
        return redirect(request.args.get("next", "/"))  # 4. open redirect
    return "invalid", 401

app.run(debug=True)                          # 5. Werkzeug debugger = remote shell

Five issues in fifteen lines, and every one of them is a pattern you can find in thousands of public examples. None of them is exotic. That's exactly why a scanner catches them reliably: they're the well-known classes that SAST and secret detection were built for.

Why code review doesn't catch it

AI-written code reads well. Names are consistent, there's a docstring, and the tests pass. A reviewer skimming a 600-line pull request sees tidy code and approves it. The Perry study suggests the author is also more confident than they should be. So both of the people who could have caught the bug are less likely to look.

Volume makes it worse. If each developer now opens twice as many pull requests, reviewers get half as much time per change. The well-known defect classes are exactly the ones a tired reviewer stops checking for.

The useful mental model: treat AI output like code from a fast, confident contractor who learned from old tutorials. You wouldn't merge that without checks. Same here.

What actually helps

  1. Scan where the code is written. Run SAST and secret detection in the IDE so the developer sees the problem while the AI suggestion is still on screen, before it's committed.
  2. Block on the pull request. High and critical findings in new code should fail the check. Let legacy findings be tracked separately, so the gate stays strict without blocking everyone on day one.
  3. Give the assistant security rules, but don't rely on them. A rules file (.cursorrules, CLAUDE.md, Copilot instructions) that says "use parameterised queries, never disable TLS verification" helps. Models still drift from it.
  4. Keep the noise down. If the scanner raises 300 findings and 280 are false positives, people stop reading. With AI volume, triage matters more than ever. AquilaX's Securitron AI filters findings before a developer sees them.
  5. Measure it. Track findings per pull request for AI-heavy repositories. If the number climbs, you know where to spend review time.

Catch the predictable mistakes

AquilaX runs SAST, secret detection, SCA and more on every commit and pull request, with AI triage so developers only see real issues.

See Vibe Code scanning →