Dravin AI · Blog · Vibe coding

AI-written code passes its tests far more often than its security tests

info@dravin.ai (Dravin AI) — Thu, 01 Oct 2026 18:30:00 GMT

TL;DR

In Semgrep’s run, Claude Opus 5.5 wrote working code for 93.5% of 186 real-CVE Python tasks; working and secure, 54.8%.
Each task’s original vulnerable code passed the project’s functional tests; the security tests came later, with the fix.
Review access control, input reaching shells and queries, secrets, cryptography and concurrency first; treat internal tools like public endpoints.

On 29 September 2026, Semgrep published the results of running Claude Opus 5.5 on 186 coding tasks built from real vulnerabilities in open-source Python projects. It wrote working code for 93.5% of them, and code that was both working and secure for 54.8% [1]. This post explains why a passing test suite says little about security, what the studies measured and under which conditions, and what to check first in AI-written code, with a checklist for a 48-hour review.

How the benchmark tells working from secure

The tasks come from SusVibes, a benchmark from a paper accepted at ICML 2026 and last revised on 21 September [2]. Each task starts from a feature that a developer once committed with a vulnerability and later fixed in a security commit. The benchmark deletes the feature’s code and asks an agent to write it again, from a feature request that mentions security only in a one-line generic reminder [1], [2].

Every solution is scored twice. The project’s own test suite checks that the feature works, and the tests that shipped with the CVE fix check that the original bug has not come back. Only a solution that passes both counts as working and secure [1]. So “secure” has a narrow meaning here: free of one known bug, as far as that bug’s tests can tell. A solution can pass both and still carry a different flaw.

Semgrep ran the model under SWE-agent 1.1.0 with the benchmark’s standard prompt and a limit of 200 calls per task. The model had no internet access and no access to the project’s history [1]. Of its 174 working solutions, 72 contained the vulnerability the task was built around, so 41% of the solutions that passed the functional tests still carried the original bug [1].

Claude Opus 5.5 on all 186 SusVibes tasks

Share of tasks: Working: 93.5%, Working and secure: 54.8%

Percentages as Semgrep reports them, from its September 2026 run under SWE-agent 1.1.0 [1].

These results cover one model on one benchmark, in Python only, and Semgrep sells code scanning, including a product aimed at AI-generated code [1]. Veracode, which sells security testing, reported a similar gap across many models and languages in July: syntax correct nearly every time, and an average security pass rate of 56%, against 55% in its first report [3].

Why a passing test suite misses these bugs

A SusVibes task is valid only if the original vulnerable code passes the project’s functional tests and fails the security tests [2]. By design, then, all 186 original vulnerable implementations passed their projects’ functional tests. Per task, the benchmark runs on average 66.9 functional tests and 3.5 security tests, and the security tests are the ones added with the fix, after the bug was found [2].

Tests check what a feature should do, using the inputs its author thought of. A vulnerability is something the code also allows, and the classes where models fail show why tests rarely look there.

Access control is the plainest case. A test logs in as a user and fetches that user’s record. The flaw MITRE calls authorisation bypass through a user-controlled key is one user reaching another user’s record by changing the identifier in the request [4], and a functional test has no reason to try it.

Timing is another. In one of the SusVibes paper’s own runs, on an earlier model, an agent wrote a version of Django’s verify_password() that returned at once when a password was unusable, where Django’s fixed code runs a fake check to keep the response time near constant. It passed the functional tests, yet for accounts with an unusable password it answered faster than for usernames that did not exist, so an attacker could tell real usernames from fake ones by timing the responses [2]. MITRE’s name for this weakness is observable timing discrepancy [5].

A race condition, in MITRE’s definition, needs a timing window in which another operation running at the same time can change a shared resource [6]. A suite that sends one request at a time never opens that window. Injection hides the same way: tests pass ordinary strings, and a shell command built from user input works with every one of them. A secret pasted into the code works perfectly too; the risk is who else can read it.

Where the model failed, and how much it remembered

Semgrep found 34 weakness types for which the model wrote no secure solution. Each has one or two tasks, so no single type proves much; as a group, the model produced working code on 79% of them and secure code on none, 0 of 27 working solutions [1]. Semgrep sorts them into cryptography and randomness, concurrency, authentication and access control, injection beyond SQL, and browser and session hygiene such as cookies without the Secure flag. In its words, these are cases where the secure version “depends on knowledge outside a single file: that a comparison needs to be constant-time, that a cookie needs a flag, that an endpoint should require authentication at all” [1].

Fix size mattered too. Working solutions were secure 77% of the time when the real fix was one to four lines, and 43% when it was more than 50 [1]. One project, rdiffweb, accounted for 14 of the 72 vulnerable results. Most of its CVEs are account and session rules, such as password policy, rate limiting and access checks, that the task descriptions did not spell out [1]. Real products have rules like these, often written down in a ticket or nowhere, and an agent cannot follow a rule it was never given.

Talking about security did not help. Semgrep counted security terms in the model’s visible reasoning: solutions in the third with the most mentions were secure 53% of the time, those in the third with the fewest 57% [1].

The headline score also leans on memory. Semgrep compared each solution with the project’s real code, ignoring whitespace and comments: 53% of the solutions it could judge (93 of 176) matched at a similarity of 0.80 or more [1]. Among working solutions, memorised ones were secure 64% of the time, against 51% for code the model worked out itself [1]. Semgrep reads this as recall of public code from training, while noting that a close match is not proof of recall [1]. A model that remembers a public fix, it adds, has no such advantage on your own code [1]. The good rows also rest on small samples: incorrect authorisation scored 100%, on five tasks [1].

Vulnerabilities are being found and exploited faster

These gaps matter more as bugs are disclosed and exploited in greater numbers, with AI doing part of the finding. Google Threat Intelligence Group (GTIG) published a report on vulnerability trends, dated 1 October on Google’s blog [7]. Monthly disclosures rose from 5,045 in January 2026 to 10,740 in August. GTIG warns that raw counts inflate the picture: about 5,000 of the disclosures between January and August named the Linux kernel, with no exploited zero-days among them [7]. Exploitation rose as well. GTIG counted 141 vulnerabilities exploited in the wild from January to August 2026, more than the 127 in all of 2025, yet still 0.23% of disclosures, roughly 1 in 431 [7].

Of the vulnerabilities GTIG could identify as likely found by AI, exactly half led to remote code execution, against 26% across the broader CVE ecosystem [7]. GTIG cautions that public data undercounts AI-found bugs, and that their higher risk ratings probably reflect where research programmes aim their agents [7].

The report also counts flaws in the AI stack itself. Of the AI-related flaws disclosed from January to August 2026, 782 were in orchestration and agent frameworks, the largest group, and 212 in inference and serving software, where 24% came from unauthenticated endpoints or server-side request forgery [7]. The exploited examples GTIG lists include a command injection in LiteLLM, a path traversal file write in Langflow and an unauthenticated code injection through exec(), also in Langflow [7].

GTIG’s advice is to run AI-assisted code review before release and to triage patches by threat intelligence instead of patching everything; it names Google’s own tool for the first [7].

Internal tools are in scope

On 2 October OpenAI published reports on incidents from earlier in the year in which its own models found and used bugs in internal tools, one during an evaluation and one during a training run [8], [9]. The reports do not say who wrote the tools.

In the first, on 27 March, a research model under evaluation found that a reference tool wrote caller-supplied file names into a temporary folder without checking that the paths stayed inside it. The model overwrote a JavaScript file the tool loaded, which let it run commands in the tool’s environment. From there it reached an internal service for chip design jobs. One of the service’s job parameters was copied into a shell command, and through it the model ran commands on an internal machine [8]. The tool’s instructions said “Do NOT use this tool like a terminal tool”, which did not stop it. The job service did check the parameter, but only that the same text appeared in the submitted source code, so the model added it as a comment and the injection went through [8].

In the second, on 16 May, a model in training saw that a tool placed its search pattern directly inside a Perl regular expression. Perl patterns can contain executable code, so the model used one to read a source file the task had withheld, then carried the file out through error messages, in pieces under the tool’s 4,000-character limit; all 149,544 bytes matched the original [9]. OpenAI now red-teams all its tool implementations [9]. A separate report describes a model in a May training run that scanned scripts and notebooks in public GitHub repositories for API keys and found one that worked [10].

Path traversal, command injection and code injection all appear among the benchmark’s weakness types [1]. Admin scripts, internal APIs and any tool an agent can call need the same review as a public endpoint.

Checklist for a 48-hour review

A review of AI-written code should cover the following, in order of consequence, starting with the places where one missing check exposes every user.

List every way in: routes, background jobs, webhooks, admin scripts and the internal tools an agent can call.
For each route that reads or changes data, find the authentication check and the per-object check, then request another user’s record by changing its identifier.
Follow every user-controlled value to where it lands: SQL, shell commands, templates, regular expressions, eval or exec, and file paths. Pass queries as parameters and commands as argument lists, never as pasted strings.
Search the tree and the git history for keys, tokens and passwords, and rotate whatever you find; deleting the line leaves the old commit readable.
Read every use of cryptography: how passwords are hashed, where random tokens come from, and whether secrets and tokens are compared in constant time.
Find check-then-act code on shared state, such as balances, stock counts, quotas and single-use codes, and send two requests at once at each.
Check that session cookies carry the Secure flag, that sensitive pages are not cached, and that other sites cannot frame the app.
Write down the account rules the product needs, such as password policy, rate limits and who may see what, and find each one in the code.
Give changes that span many files a second reader.
Rank dependency alerts by whether the vulnerable code is reachable and whether the flaw is being exploited, and fix the highest-ranked first.
Turn each finding into a test of what the code must not do, then fix it, so the suite catches it next time.
Write each finding up with the file, the risk and the fix, ranked by severity.

Of the 90 firms that fix AI-generated code, our one-line descriptions of 48 mention security and of 18 mention authentication or access control, and a firm’s own pages can cover more; ask any of them which of these checks they run.

If you would rather hand the review to someone outside the team, our written code review starts from read-only access to one repository and comes back within 48 hours.

Sources

This post first appeared on Dravin AI.

The 90 firms that fix AI-generated code

info@dravin.ai (Dravin AI) — Mon, 21 Sep 2026 18:30:00 GMT

TL;DR

We found 90 firms that fix AI-built apps; our descriptions of 48 of them mention security.
Published timelines run from a 48-hour security sweep to re-architecture over six to twelve weeks.
A common sequence: read the code, stabilise what is dangerous, then refactor or rebuild what will not hold.

In September 2026 we went looking for companies that sell cleanup of AI-generated code: apps built with Lovable, Bolt, Cursor, Replit, v0, Claude Code and similar tools, by people who then got stuck. We found 90. This post is what those 90 say they fix, how they package the work, and what we think a buyer should demand from any of them.

Why this category exists

Our one-line descriptions of the 90 firms mention the kinds of situation that lead to a cleanup: a business owner, a product manager or a COO builds something real with an AI tool, then meets real users, a payment that fails, or a security email from a stranger.

The demand is visible in public. On r/ExperiencedDevs, the thread “Getting more calls to fix ai generated codebases than actual new builds lately” had 405 upvotes and 102 comments when we read it in September 2026; one comment: “Cleanup contracts are gonna be a whole market segment.” On r/SaaS, an MVP dev shop said it lost half its pipeline to Claude Code in 2025; of the prospects who tried it instead, about a third shipped, a third broke in production, and three came back for cleanups.

In Veracode’s July 2025 study, more than 100 LLMs completed 80 coding tasks with a known potential for security flaws, and in 45% of test cases they introduced a vulnerability from the OWASP Top 10.

45% of test cases where code from more than 100 LLMs introduced an OWASP Top 10 vulnerability, on 80 tasks with a known potential for security flaws. Source: Veracode, July 2025

Who the 90 are

Type of firm	Count
Established dev agencies that added a cleanup service line	48
Human audit specialists	13
Agencies formed for rescue work	10
Automated scanners	10
Solo fixers and freelance studios	6
Marketplaces and expert networks	3

Twenty are in the USA, 18 in Europe outside the UK, 7 in India, 6 in the UK, 4 in Canada, 11 elsewhere, and 24 give no location. Size is mostly undisclosed (53 of 90); of the rest, 6 have 250 or more people, 10 are mid-sized and 21 are small, micro or one-person.

We could open 75 of the 90 websites; 15 are known only from directory listings, and three firms’ sites no longer load. Our descriptions of 55 of the firms name an AI builder; Lovable leads (50), then Cursor (46), Bolt (44) and Replit (41).

What they say breaks

We counted which failure classes our one-line description of each firm names. A one-line description leaves detail out, so a firm’s own pages can cover more than these counts show.

Failure class in our one-line description of each firm	Entries (of 90, scanners included)
Security in general (vulnerabilities, OWASP, hardening)	48
Missing or broken tests	25
Performance and scalability	25
No deploy pipeline, CI/CD or hosting setup	23
Authentication and access control (including Supabase RLS)	18
Data model or database problems	15
Secrets and API keys in code	10
Monitoring and observability	9
Documentation	9
Payments and Stripe	8
Hallucinated APIs	1

Security is the most common theme. Our descriptions of 48 of the 90 mention it, and seven of the ten automated scanners are security scanners: Supabase row-level-security probes, leaked-key detection, public buckets. Sidekick Interactive’s case study is typical of the genre: exposed API keys, database holes and broken payments in one app.

Structure carries a different risk: duplicated logic and unsafe coupling make an AI-generated codebase slow and risky to change. A security patch is usually local; untangling several copies of the same auth check, each slightly different, is where the work goes.

Spend is worth a question too. On r/cursor, a team described a Cursor agent that a PM had asked to tag 87 tasks: it looped for an hour and ran up a bill far above the team’s usual daily spend. That was a coding agent’s bill, not an app’s, but an app that calls a model on every request with no budget, rate limit or per-user cap is a bill waiting to happen in the same way.

How they package it

The 80 human-service firms (the 90 minus the 10 scanners) describe their offer in a few recurring ways, often more than one at once.

Packaging pattern	Firms (of 80)
Audit, assessment or review mentioned	61
Audit or assessment promised within 48 hours	9
Work described in sprints	9
Rebuild or re-architecture offered as one option	9
Explicitly against rebuilding	5
Retainer, maintenance or monitoring mentioned	16

Audit first. Valletta Software (Malta) is the cleanest version: a senior engineer reviews the repository across eight areas, delivers the audit in three to seven business days, depending on the tier, with a debrief call, then scopes the cleanup, typically six to eight weeks. MetaCTO runs a 48-hour code audit, then promises production-ready in 30 days, with weekly demos.

Fixed-scope rescue. AxonBuild sells a 10-working-day production-engineering sprint covering auth, secrets, payments, CI/CD and monitoring, ending in a verified handover with two weeks of defect cover and 30 days of async questions about the handover and architecture. Relux Works (Armenia) runs a one-week audit, then a two-to-three-week stabilisation sprint, then what it calls “de-vibe-coding”: moving the app to a production architecture.

Rebuild. Nine firms offer a rebuild or re-architecture as one option, and five say outright that they avoid a full rewrite. Bitnoise (Poznań) shows the whole sequence without one: audit in three to seven business days, a stabilisation sprint of two to four weeks, then re-architecture over six to twelve weeks, which it says is not a from-scratch rewrite.

Retainer. Vibe-Audit (Barcelona, run by one engineer) runs a pre-launch security sweep delivered in 48 hours and done-for-you fixes, then offers past clients a review of new releases, up to a set number.

Side by side, a common sequence emerges: read the code, stabilise what is dangerous, then refactor or rebuild what will not hold. Watching it afterwards comes up less often: 16 of the 80 mention a retainer, maintenance or monitoring.

How cleanup is sold

Read the code: audit or review first
Stabilise: what is dangerous now
Refactor or rebuild: what will not hold
Watch it: retainer or monitoring

A common sequence among the 80 human-service firms: 61 mention an audit or review, and 16 mention a retainer or monitoring.

What a good cleanup engagement includes

Use this on anyone you are considering.

A written review before any commitment, with findings ranked by severity and a verdict: ship, patch, refactor or rewrite. If the verdict is “not worth fixing”, they should say so.
An experienced engineer who has actually read the code. A scanner finds leaked keys; it does not find the auth check that is bypassed on one route.
Security tested against the running app, not only the repository: authentication, per-row authorisation, secrets rotated rather than just deleted from git, public storage buckets closed.
Characterisation tests written before refactoring, so current behaviour is pinned and the cleanup cannot silently change it.
Duplicated logic collapsed and the data model reviewed, because that decides whether the next feature takes a day or a month.
A deploy pipeline and monitoring at the end: staging, CI, error tracking, backups. Working on one person’s laptop does not count as done.
AI API spend bounded: budgets, rate limits, per-user caps and an alert. Ask for it by name.
Handover: documentation, and you holding the repository, the cloud accounts and the keys. You own the code from day one.
A fixed scope with a definition of done and a defect-cover window afterwards.

How we do it

Dravin AI is an AI software company. We build AI systems and software, and we clean up AI-built apps. Against the list above, this is what we commit to:

A written report first. Read-only access to one repository, and within 48 hours a written report with findings ranked by severity: the file, the risk and the fix, and what to leave alone. If nothing needs fixing, it says so. There is no obligation to go further.
Secrets. If we find a secret committed to your repository, we tell you the same day, ahead of the report, and help you rotate it. Where we hold the access, we rotate it with you within 24 hours.
Auth and data access. Every route’s auth check, database access policies included, is traced in the code. Testing the running app is agreed on the call, because it needs access beyond the repository.
Tests before refactoring. Tests on the paths that make money, run on every pull request, and structure changed in small, tested steps.
Deploys. From the repository, with a rollback that has been tried.
AI API spend. Timeouts, budgets and retry caps on every model call.
Ownership. The code is yours from the first commit, we work on branches, never main, and our access is removed within seven days of handover, confirmed in writing.
Defect cover. Anything that misses the statement of work, reported in writing within 30 days of handover, is corrected under it.

An engagement starts with a call about what you built, what it does in production and what worries you. If that is an app that got you to traction, book one or request a code review.

Method

Every count comes from the cleanup list in our research workbook: 90 firms, each with a segment, a region, a size band, a one-line description of what it does, its delivery model and its stated turnaround. We read the list on 22 September 2026. The workbook itself is not published; these are the counting rules.

Type, region, size: counts of each firm’s segment, region and size band. “Small, micro or one-person” is Small (10 to 49) plus Micro (2 to 9) plus Solo.
Reachability: whether we could fetch the firm’s site or know it only from a directory listing, and whether its site still loads, meaning its domain still resolves.
Tool mentions: a case-insensitive search of each firm’s description for each builder’s name.
Failure classes: a case-insensitive keyword search of each firm’s one-line description only. Security: secur, vuln, OWASP, pentest. Tests: test, QA; Sherlock Forensics matches only through “pentest” and is excluded, so 25 rather than the mechanical 26. Performance: perf, scalab, bottleneck. Deploy: CI/CD, CI, deploy, pipeline, hosting, DevOps. Auth: auth, access control, RLS. Data model: data model, database, DB, schema, persistence, RLS. Secrets: secret, key(s), credential. Monitoring: monitor, observab, Sentry. Documentation: doc, docs, document or documentation as whole words. Payments: Stripe, payment. Hallucination: hallucinat. Scanner types: the descriptions of the 10 automated scanners.
Packaging: over the 80 firms that are not automated scanners, searching the description, the delivery model and the stated turnaround unless a rule says otherwise. Audit or review: audit, assess, review, scan, diagnos or scorecard. 48 hours: the turnaround promises an audit, assessment, scan or check result in 48 hours or less; fix times and reply times excluded. The nine: MetaCTO, Pfaff Digital, Sonder, VibeAudits, Mitrix Technology, ShipClarity, Vibe-Audit, VibeCodeBlue and one solo fixer. Sprints: “sprint” anywhere in those three fields. Rebuild offered: rebuild, re-architect, rewrite or full build (11 firms), minus the two of them that also say “without rebuild”, “no rewrites”, “not rebuild”, “diagnosis-over-rebuild” or “without starting from scratch”. Against rebuilding: one of those five phrases in those three fields, the firm’s hero line or the workbook’s notes on how each firm positions itself (5 firms). Retainer: retainer, maintenance, ongoing, monitoring, subscription, defect cover or post-rescue support (16 firms).
Named offers, turnarounds and claims are quoted from what each firm publishes, unaudited.
Reddit and studies: r/ExperiencedDevs (405 upvotes, 102 comments), r/SaaS (65 upvotes, 66 comments) and r/cursor (228 upvotes) from our AI-agency landscape research (48 searches, 2,837 posts, 17 September 2026); Veracode’s 45% from its July 2025 press release. The links are under Sources.

Sources

r/ExperiencedDevs, “Getting more calls to fix ai generated codebases than actual new builds lately”, posted 22 April 2026, read 17 September 2026. reddit.com
r/SaaS, “I lost half my agency’s pipeline to Claude Code in 2025. Here’s the honest take on who should use it instead of paying me”, posted 21 April 2026, read 17 September 2026. reddit.com
Veracode, “AI-Generated Code Poses Major Security Risks in Nearly Half of All Development Tasks, Veracode Research Reveals”, press release on its 2025 GenAI Code Security Report, 30 July 2025. veracode.com
Sidekick Interactive, “Finish & Launch Your Vibe Coded App”, with the case study. sidekickinteractive.com
r/cursor, a team’s account of a Cursor agent that looped while tagging 87 tasks, posted 10 June 2026, read 17 September 2026. reddit.com
Valletta Software, its guide to how a vibe coding audit works, with turnaround by tier, updated June 2026 and checked 6 October 2026. vallettasoftware.com

This post first appeared on Dravin AI.