AI-written code passes its tests far more often than its security tests

On 29 September 2026, Semgrep published the results of running Claude Opus 5.5 on 186 coding tasks built from real vulnerabilities in open-source Python projects. It wrote working code for 93.5% of them, and code that was both working and secure for 54.8% [1]. This post explains why a passing test suite says little about security, what the studies measured and under which conditions, and what to check first in AI-written code, with a checklist for a 48-hour review.

How the benchmark tells working from secure

The tasks come from SusVibes, a benchmark from a paper accepted at ICML 2026 and last revised on 21 September [2]. Each task starts from a feature that a developer once committed with a vulnerability and later fixed in a security commit. The benchmark deletes the feature’s code and asks an agent to write it again, from a feature request that mentions security only in a one-line generic reminder [1], [2].

Every solution is scored twice. The project’s own test suite checks that the feature works, and the tests that shipped with the CVE fix check that the original bug has not come back. Only a solution that passes both counts as working and secure [1]. So “secure” has a narrow meaning here: free of one known bug, as far as that bug’s tests can tell. A solution can pass both and still carry a different flaw.

Semgrep ran the model under SWE-agent 1.1.0 with the benchmark’s standard prompt and a limit of 200 calls per task. The model had no internet access and no access to the project’s history [1]. Of its 174 working solutions, 72 contained the vulnerability the task was built around, so 41% of the solutions that passed the functional tests still carried the original bug [1].

Claude Opus 5.5 on all 186 SusVibes tasks

WorkingWorking and secure

• Share of tasks:
  Working: 93.5%, Working and secure: 54.8%

These results cover one model on one benchmark, in Python only, and Semgrep sells code scanning, including a product aimed at AI-generated code [1]. Veracode, which sells security testing, reported a similar gap across many models and languages in July: syntax correct nearly every time, and an average security pass rate of 56%, against 55% in its first report [3].

Why a passing test suite misses these bugs

A SusVibes task is valid only if the original vulnerable code passes the project’s functional tests and fails the security tests [2]. By design, then, all 186 original vulnerable implementations passed their projects’ functional tests. Per task, the benchmark runs on average 66.9 functional tests and 3.5 security tests, and the security tests are the ones added with the fix, after the bug was found [2].

Tests check what a feature should do, using the inputs its author thought of. A vulnerability is something the code also allows, and the classes where models fail show why tests rarely look there.

Access control is the plainest case. A test logs in as a user and fetches that user’s record. The flaw MITRE calls authorisation bypass through a user-controlled key is one user reaching another user’s record by changing the identifier in the request [4], and a functional test has no reason to try it.

Timing is another. In one of the SusVibes paper’s own runs, on an earlier model, an agent wrote a version of Django’s verify_password() that returned at once when a password was unusable, where Django’s fixed code runs a fake check to keep the response time near constant. It passed the functional tests, yet for accounts with an unusable password it answered faster than for usernames that did not exist, so an attacker could tell real usernames from fake ones by timing the responses [2]. MITRE’s name for this weakness is observable timing discrepancy [5].

A race condition, in MITRE’s definition, needs a timing window in which another operation running at the same time can change a shared resource [6]. A suite that sends one request at a time never opens that window. Injection hides the same way: tests pass ordinary strings, and a shell command built from user input works with every one of them. A secret pasted into the code works perfectly too; the risk is who else can read it.

Enjoying this? New posts by email

Leave this field empty

Email

SubscribeSubscribing

Where the model failed, and how much it remembered

Semgrep found 34 weakness types for which the model wrote no secure solution. Each has one or two tasks, so no single type proves much; as a group, the model produced working code on 79% of them and secure code on none, 0 of 27 working solutions [1]. Semgrep sorts them into cryptography and randomness, concurrency, authentication and access control, injection beyond SQL, and browser and session hygiene such as cookies without the Secure flag. In its words, these are cases where the secure version “depends on knowledge outside a single file: that a comparison needs to be constant-time, that a cookie needs a flag, that an endpoint should require authentication at all” [1].

Fix size mattered too. Working solutions were secure 77% of the time when the real fix was one to four lines, and 43% when it was more than 50 [1]. One project, rdiffweb, accounted for 14 of the 72 vulnerable results. Most of its CVEs are account and session rules, such as password policy, rate limiting and access checks, that the task descriptions did not spell out [1]. Real products have rules like these, often written down in a ticket or nowhere, and an agent cannot follow a rule it was never given.

Talking about security did not help. Semgrep counted security terms in the model’s visible reasoning: solutions in the third with the most mentions were secure 53% of the time, those in the third with the fewest 57% [1].

The headline score also leans on memory. Semgrep compared each solution with the project’s real code, ignoring whitespace and comments: 53% of the solutions it could judge (93 of 176) matched at a similarity of 0.80 or more [1]. Among working solutions, memorised ones were secure 64% of the time, against 51% for code the model worked out itself [1]. Semgrep reads this as recall of public code from training, while noting that a close match is not proof of recall [1]. A model that remembers a public fix, it adds, has no such advantage on your own code [1]. The good rows also rest on small samples: incorrect authorisation scored 100%, on five tasks [1].

Vulnerabilities are being found and exploited faster

These gaps matter more as bugs are disclosed and exploited in greater numbers, with AI doing part of the finding. Google Threat Intelligence Group (GTIG) published a report on vulnerability trends, dated 1 October on Google’s blog [7]. Monthly disclosures rose from 5,045 in January 2026 to 10,740 in August. GTIG warns that raw counts inflate the picture: about 5,000 of the disclosures between January and August named the Linux kernel, with no exploited zero-days among them [7]. Exploitation rose as well. GTIG counted 141 vulnerabilities exploited in the wild from January to August 2026, more than the 127 in all of 2025, yet still 0.23% of disclosures, roughly 1 in 431 [7].

Of the vulnerabilities GTIG could identify as likely found by AI, exactly half led to remote code execution, against 26% across the broader CVE ecosystem [7]. GTIG cautions that public data undercounts AI-found bugs, and that their higher risk ratings probably reflect where research programmes aim their agents [7].

The report also counts flaws in the AI stack itself. Of the AI-related flaws disclosed from January to August 2026, 782 were in orchestration and agent frameworks, the largest group, and 212 in inference and serving software, where 24% came from unauthenticated endpoints or server-side request forgery [7]. The exploited examples GTIG lists include a command injection in LiteLLM, a path traversal file write in Langflow and an unauthenticated code injection through exec(), also in Langflow [7].

GTIG’s advice is to run AI-assisted code review before release and to triage patches by threat intelligence instead of patching everything; it names Google’s own tool for the first [7].

Internal tools are in scope

On 2 October OpenAI published reports on incidents from earlier in the year in which its own models found and used bugs in internal tools, one during an evaluation and one during a training run [8], [9]. The reports do not say who wrote the tools.

In the first, on 27 March, a research model under evaluation found that a reference tool wrote caller-supplied file names into a temporary folder without checking that the paths stayed inside it. The model overwrote a JavaScript file the tool loaded, which let it run commands in the tool’s environment. From there it reached an internal service for chip design jobs. One of the service’s job parameters was copied into a shell command, and through it the model ran commands on an internal machine [8]. The tool’s instructions said “Do NOT use this tool like a terminal tool”, which did not stop it. The job service did check the parameter, but only that the same text appeared in the submitted source code, so the model added it as a comment and the injection went through [8].

In the second, on 16 May, a model in training saw that a tool placed its search pattern directly inside a Perl regular expression. Perl patterns can contain executable code, so the model used one to read a source file the task had withheld, then carried the file out through error messages, in pieces under the tool’s 4,000-character limit; all 149,544 bytes matched the original [9]. OpenAI now red-teams all its tool implementations [9]. A separate report describes a model in a May training run that scanned scripts and notebooks in public GitHub repositories for API keys and found one that worked [10].

Path traversal, command injection and code injection all appear among the benchmark’s weakness types [1]. Admin scripts, internal APIs and any tool an agent can call need the same review as a public endpoint.

Checklist for a 48-hour review

A review of AI-written code should cover the following, in order of consequence, starting with the places where one missing check exposes every user.

1. List every way in: routes, background jobs, webhooks, admin scripts and the internal tools an agent can call.
2. For each route that reads or changes data, find the authentication check and the per-object check, then request another user’s record by changing its identifier.
3. Follow every user-controlled value to where it lands: SQL, shell commands, templates, regular expressions, eval or exec, and file paths. Pass queries as parameters and commands as argument lists, never as pasted strings.
4. Search the tree and the git history for keys, tokens and passwords, and rotate whatever you find; deleting the line leaves the old commit readable.
5. Read every use of cryptography: how passwords are hashed, where random tokens come from, and whether secrets and tokens are compared in constant time.
6. Find check-then-act code on shared state, such as balances, stock counts, quotas and single-use codes, and send two requests at once at each.
7. Check that session cookies carry the Secure flag, that sensitive pages are not cached, and that other sites cannot frame the app.
8. Write down the account rules the product needs, such as password policy, rate limits and who may see what, and find each one in the code.
9. Give changes that span many files a second reader.
10. Rank dependency alerts by whether the vulnerable code is reachable and whether the flaw is being exploited, and fix the highest-ranked first.
11. Turn each finding into a test of what the code must not do, then fix it, so the suite catches it next time.
12. Write each finding up with the file, the risk and the fix, ranked by severity.

Of the 90 firms that fix AI-generated code, our one-line descriptions of 48 mention security and of 18 mention authentication or access control, and a firm’s own pages can cover more; ask any of them which of these checks they run.

If you would rather hand the review to someone outside the team, our written code review starts from read-only access to one repository and comes back within 48 hours.

Sources

1. 1Semgrep, “Claude Opus 5.5 Writes Working Code but Only Half of It Is Secure” (29 Sep 2026)
2. 2arXiv, “Is Vibe Coding Safe? Benchmarking Vulnerability of Agent-Generated Code in Real-World Tasks”, ICML 2026 (v4, 21 Sep 2026)
3. 3Veracode, “2026 GenAI Code Security Report: AI Is Writing More of Your Code but Security Hasn’t Caught Up” (28 Jul 2026)
4. 4MITRE, “CWE-639: Authorization Bypass Through User-Controlled Key” (CWE 4.20, 30 Apr 2026)
5. 5MITRE, “CWE-208: Observable Timing Discrepancy” (CWE 4.20, 30 Apr 2026)
6. 6MITRE, “CWE-362: Concurrent Execution using Shared Resource with Improper Synchronization (‘Race Condition’)” (CWE 4.20, 30 Apr 2026)
7. 7Google Threat Intelligence Group, “Vulnerability Discovery and Exploitation Trends in the AI Era” (1 Oct 2026)
8. 8OpenAI, “Reaching an internal EDA host through a reference tool” (2 Oct 2026)
9. 9OpenAI, “Command injecting a reference tool to copy a source file” (2 Oct 2026)
10. 10OpenAI, “Signing up for disposable emails and searching GitHub for leaked API keys” (16 Sep 2026)