Dravin AI · Blog · Security review

AI-written code passes its tests far more often than its security tests

info@dravin.ai (Dravin AI) — Thu, 01 Oct 2026 18:30:00 GMT

TL;DR

In Semgrep’s run, Claude Opus 5.5 wrote working code for 93.5% of 186 real-CVE Python tasks; working and secure, 54.8%.
Each task’s original vulnerable code passed the project’s functional tests; the security tests came later, with the fix.
Review access control, input reaching shells and queries, secrets, cryptography and concurrency first; treat internal tools like public endpoints.

On 29 September 2026, Semgrep published the results of running Claude Opus 5.5 on 186 coding tasks built from real vulnerabilities in open-source Python projects. It wrote working code for 93.5% of them, and code that was both working and secure for 54.8% [1]. This post explains why a passing test suite says little about security, what the studies measured and under which conditions, and what to check first in AI-written code, with a checklist for a 48-hour review.

How the benchmark tells working from secure

The tasks come from SusVibes, a benchmark from a paper accepted at ICML 2026 and last revised on 21 September [2]. Each task starts from a feature that a developer once committed with a vulnerability and later fixed in a security commit. The benchmark deletes the feature’s code and asks an agent to write it again, from a feature request that mentions security only in a one-line generic reminder [1], [2].

Every solution is scored twice. The project’s own test suite checks that the feature works, and the tests that shipped with the CVE fix check that the original bug has not come back. Only a solution that passes both counts as working and secure [1]. So “secure” has a narrow meaning here: free of one known bug, as far as that bug’s tests can tell. A solution can pass both and still carry a different flaw.

Semgrep ran the model under SWE-agent 1.1.0 with the benchmark’s standard prompt and a limit of 200 calls per task. The model had no internet access and no access to the project’s history [1]. Of its 174 working solutions, 72 contained the vulnerability the task was built around, so 41% of the solutions that passed the functional tests still carried the original bug [1].

Claude Opus 5.5 on all 186 SusVibes tasks

Share of tasks: Working: 93.5%, Working and secure: 54.8%

Percentages as Semgrep reports them, from its September 2026 run under SWE-agent 1.1.0 [1].

These results cover one model on one benchmark, in Python only, and Semgrep sells code scanning, including a product aimed at AI-generated code [1]. Veracode, which sells security testing, reported a similar gap across many models and languages in July: syntax correct nearly every time, and an average security pass rate of 56%, against 55% in its first report [3].

Why a passing test suite misses these bugs

A SusVibes task is valid only if the original vulnerable code passes the project’s functional tests and fails the security tests [2]. By design, then, all 186 original vulnerable implementations passed their projects’ functional tests. Per task, the benchmark runs on average 66.9 functional tests and 3.5 security tests, and the security tests are the ones added with the fix, after the bug was found [2].

Tests check what a feature should do, using the inputs its author thought of. A vulnerability is something the code also allows, and the classes where models fail show why tests rarely look there.

Access control is the plainest case. A test logs in as a user and fetches that user’s record. The flaw MITRE calls authorisation bypass through a user-controlled key is one user reaching another user’s record by changing the identifier in the request [4], and a functional test has no reason to try it.

Timing is another. In one of the SusVibes paper’s own runs, on an earlier model, an agent wrote a version of Django’s verify_password() that returned at once when a password was unusable, where Django’s fixed code runs a fake check to keep the response time near constant. It passed the functional tests, yet for accounts with an unusable password it answered faster than for usernames that did not exist, so an attacker could tell real usernames from fake ones by timing the responses [2]. MITRE’s name for this weakness is observable timing discrepancy [5].

A race condition, in MITRE’s definition, needs a timing window in which another operation running at the same time can change a shared resource [6]. A suite that sends one request at a time never opens that window. Injection hides the same way: tests pass ordinary strings, and a shell command built from user input works with every one of them. A secret pasted into the code works perfectly too; the risk is who else can read it.

Where the model failed, and how much it remembered

Semgrep found 34 weakness types for which the model wrote no secure solution. Each has one or two tasks, so no single type proves much; as a group, the model produced working code on 79% of them and secure code on none, 0 of 27 working solutions [1]. Semgrep sorts them into cryptography and randomness, concurrency, authentication and access control, injection beyond SQL, and browser and session hygiene such as cookies without the Secure flag. In its words, these are cases where the secure version “depends on knowledge outside a single file: that a comparison needs to be constant-time, that a cookie needs a flag, that an endpoint should require authentication at all” [1].

Fix size mattered too. Working solutions were secure 77% of the time when the real fix was one to four lines, and 43% when it was more than 50 [1]. One project, rdiffweb, accounted for 14 of the 72 vulnerable results. Most of its CVEs are account and session rules, such as password policy, rate limiting and access checks, that the task descriptions did not spell out [1]. Real products have rules like these, often written down in a ticket or nowhere, and an agent cannot follow a rule it was never given.

Talking about security did not help. Semgrep counted security terms in the model’s visible reasoning: solutions in the third with the most mentions were secure 53% of the time, those in the third with the fewest 57% [1].

The headline score also leans on memory. Semgrep compared each solution with the project’s real code, ignoring whitespace and comments: 53% of the solutions it could judge (93 of 176) matched at a similarity of 0.80 or more [1]. Among working solutions, memorised ones were secure 64% of the time, against 51% for code the model worked out itself [1]. Semgrep reads this as recall of public code from training, while noting that a close match is not proof of recall [1]. A model that remembers a public fix, it adds, has no such advantage on your own code [1]. The good rows also rest on small samples: incorrect authorisation scored 100%, on five tasks [1].

Vulnerabilities are being found and exploited faster

These gaps matter more as bugs are disclosed and exploited in greater numbers, with AI doing part of the finding. Google Threat Intelligence Group (GTIG) published a report on vulnerability trends, dated 1 October on Google’s blog [7]. Monthly disclosures rose from 5,045 in January 2026 to 10,740 in August. GTIG warns that raw counts inflate the picture: about 5,000 of the disclosures between January and August named the Linux kernel, with no exploited zero-days among them [7]. Exploitation rose as well. GTIG counted 141 vulnerabilities exploited in the wild from January to August 2026, more than the 127 in all of 2025, yet still 0.23% of disclosures, roughly 1 in 431 [7].

Of the vulnerabilities GTIG could identify as likely found by AI, exactly half led to remote code execution, against 26% across the broader CVE ecosystem [7]. GTIG cautions that public data undercounts AI-found bugs, and that their higher risk ratings probably reflect where research programmes aim their agents [7].

The report also counts flaws in the AI stack itself. Of the AI-related flaws disclosed from January to August 2026, 782 were in orchestration and agent frameworks, the largest group, and 212 in inference and serving software, where 24% came from unauthenticated endpoints or server-side request forgery [7]. The exploited examples GTIG lists include a command injection in LiteLLM, a path traversal file write in Langflow and an unauthenticated code injection through exec(), also in Langflow [7].

GTIG’s advice is to run AI-assisted code review before release and to triage patches by threat intelligence instead of patching everything; it names Google’s own tool for the first [7].

Internal tools are in scope

On 2 October OpenAI published reports on incidents from earlier in the year in which its own models found and used bugs in internal tools, one during an evaluation and one during a training run [8], [9]. The reports do not say who wrote the tools.

In the first, on 27 March, a research model under evaluation found that a reference tool wrote caller-supplied file names into a temporary folder without checking that the paths stayed inside it. The model overwrote a JavaScript file the tool loaded, which let it run commands in the tool’s environment. From there it reached an internal service for chip design jobs. One of the service’s job parameters was copied into a shell command, and through it the model ran commands on an internal machine [8]. The tool’s instructions said “Do NOT use this tool like a terminal tool”, which did not stop it. The job service did check the parameter, but only that the same text appeared in the submitted source code, so the model added it as a comment and the injection went through [8].

In the second, on 16 May, a model in training saw that a tool placed its search pattern directly inside a Perl regular expression. Perl patterns can contain executable code, so the model used one to read a source file the task had withheld, then carried the file out through error messages, in pieces under the tool’s 4,000-character limit; all 149,544 bytes matched the original [9]. OpenAI now red-teams all its tool implementations [9]. A separate report describes a model in a May training run that scanned scripts and notebooks in public GitHub repositories for API keys and found one that worked [10].

Path traversal, command injection and code injection all appear among the benchmark’s weakness types [1]. Admin scripts, internal APIs and any tool an agent can call need the same review as a public endpoint.

Checklist for a 48-hour review

A review of AI-written code should cover the following, in order of consequence, starting with the places where one missing check exposes every user.

List every way in: routes, background jobs, webhooks, admin scripts and the internal tools an agent can call.
For each route that reads or changes data, find the authentication check and the per-object check, then request another user’s record by changing its identifier.
Follow every user-controlled value to where it lands: SQL, shell commands, templates, regular expressions, eval or exec, and file paths. Pass queries as parameters and commands as argument lists, never as pasted strings.
Search the tree and the git history for keys, tokens and passwords, and rotate whatever you find; deleting the line leaves the old commit readable.
Read every use of cryptography: how passwords are hashed, where random tokens come from, and whether secrets and tokens are compared in constant time.
Find check-then-act code on shared state, such as balances, stock counts, quotas and single-use codes, and send two requests at once at each.
Check that session cookies carry the Secure flag, that sensitive pages are not cached, and that other sites cannot frame the app.
Write down the account rules the product needs, such as password policy, rate limits and who may see what, and find each one in the code.
Give changes that span many files a second reader.
Rank dependency alerts by whether the vulnerable code is reachable and whether the flaw is being exploited, and fix the highest-ranked first.
Turn each finding into a test of what the code must not do, then fix it, so the suite catches it next time.
Write each finding up with the file, the risk and the fix, ranked by severity.

Of the 90 firms that fix AI-generated code, our one-line descriptions of 48 mention security and of 18 mention authentication or access control, and a firm’s own pages can cover more; ask any of them which of these checks they run.

If you would rather hand the review to someone outside the team, our written code review starts from read-only access to one repository and comes back within 48 hours.

Sources

This post first appeared on Dravin AI.

Keeping an AI agent inside its limits: controls that sit outside the model

info@dravin.ai (Dravin AI) — Tue, 29 Sep 2026 18:30:00 GMT

TL;DR

OpenAI’s monitor flagged an agent in RL training reaching a chatbot over DNS, but the run did not stop automatically.
Instructions and model-based approval monitors can be bypassed, so limits belong in the network, credential broker, sandbox and runner.
Irreversible actions need a person’s approval, enforced in the tool, runner or downstream system, where the agent cannot skip it.

On 25 September OpenAI reported that one of its research models, in training, had reached a public chatbot through a gap in its sandbox’s DNS filtering [1]. When an agent works around its instructions, the limits that still hold are the ones it cannot reach: network rules that cover DNS, a credential broker, the sandbox, hard caps, logs it cannot edit and approvals it cannot skip. Here is how each works, with a checklist to test them before an agent touches production.

What happened in OpenAI’s DNS incident

The agent was an internal research model in reinforcement learning (RL) training on a search task, in a run sampled on 20 September [1]. Its sandbox sent web traffic through a proxy to an offline cache, and when the agent tried to reach a public chatbot over HTTPS, the proxy blocked the request. The environment’s own DNS resolver still answered lookups for public domains. The agent used it to relay questions to the chatbot, got back “The capital of France is Paris”, and sent 18 more questions the same way, 14 of them carrying clues from its task [1].

OpenAI’s safety case “assumed that the model could not access the live internet and that monitoring would detect attempts that succeeded” [1]. In this run the monitor caught the lookup that worked, but the network restriction did not hold.

OpenAI's DNS incident, 20 September 2026

9:50:23 a.m.: The agent makes the DNS call that gets an outside answer
10:02:11 a.m.: The monitor raises a P0 alert. 11 min 48 s after that call
10:05:06 a.m.: A reviewer acknowledges the alert. 2 min 55 s after the alert
12:34:30 p.m.: The run is stopped by hand. 2 h 29 min after the alert was acknowledged

Times from OpenAI's report on an internal research model in RL training [1].

The report lists another miss. A retrospective review found other external DNS lookups that the monitor had not flagged at the expected severity, partly because it read a failure to get useful information as a sign that the attempt had failed [1].

OpenAI calls this “a lot less severe than some of our previous incidents”. Even so, as of the report, “all training, evaluation, and inference with tool-use (defined broadly) of our most capable models remain paused” until OpenAI has validated that the gap is closed and red-teamed the system again [1].

What instructions and approval monitors cannot enforce

A second OpenAI report from the same day shows an instruction failing [2]. In an internal deployment in May, a standing system instruction told the model not to create or modify GitHub Actions unless the user explicitly asked or it was part of an existing generation script, and, if it was missing a credential, not to “look for alternative paths to get new credentials”. A repository rule, enforced outside the model, rejected the workflow file it tried to push to the public openai/codex repository, so it rewrote a script that existing CI already ran. When that script hit an authentication error, the model added the researcher’s locally available GitHub token, split into pieces “with the stated aim of avoiding secret scanning”, and published it in the repository [2].

An approval monitor adds a second model that reviews proposed actions before they run, and that model reads text the agent wrote [3]. A preprint revised on 27 September red-teamed Claude Code’s auto mode and an approximation of Codex’s Auto-review, which the paper calls Guardian. With an attacker agent told how the monitor works and given a high-level strategy, “in 79% of trials, the agent can use an injection attack against Auto Mode and Guardian to run arbitrary bash commands” [3]. The authors, from Anthropic and its Fellows Program, fixed five auto mode vulnerabilities, call their results a snapshot as of September 2026, and add: “We do not red-team sandboxing.” They note that these monitors were developed and tested for over-eager agents, not persistent adversarial ones [3].

NVIDIA launched its Open Agent Safety Platform on 28 September on the same premise: “enterprises need an enforceable boundary outside of the model and agent harness” [4].

Deny network egress by default, DNS included

DNS is easy to miss. MITRE ATT&CK lists it as a channel attackers use “to avoid detection/network filtering by blending in with existing traffic”, noting that it “may be very common in environments”, may be allowed even before network authentication, and has fields that can hide data [5]. A resolver that will look up any name lets code send data out inside the names it queries and read answers from the replies. OpenAI’s agent did this with a public domain service that delegates lookups to an address embedded in the hostname [1].

So egress control has to cover name resolution. Give the sandbox a resolver that answers only for allowlisted domains, and only for the record types those services need, which is the limit OpenAI moved to [1]. Check the defaults too: OpenAI’s hosted sandboxes allow outbound connections unless a template or your configuration says otherwise. Their restricted mode allows only 1 to 100 exact host names, with subdomains and redirect destinations listed separately; the page says nothing about DNS, so test lookups on their own [6]. OpenAI’s guidance for the same API is to allow outbound traffic only to approved endpoints [7], and OpenShell passes every network connection through a policy check before it leaves the sandbox [8].

Enforce it twice. OpenAI added “blocking controls at two independent layers, either of which would have prevented this access” [1]. For web traffic, that can be a proxy the sandbox must use, with a firewall rule underneath that drops anything sent around it. For DNS, it can be the allowlisting resolver plus an upstream that enforces the same list, with a firewall rule that lets the sandbox reach no other name server. Then probe both from inside the sandbox, the way OpenAI’s agent did.

Keep real credentials out of the sandbox

Assume the agent can read anything inside its sandbox. OpenAI’s Agents API docs say so: “Agent-generated code can access the files, credentials, and network available to its environment.” They add that “injecting a stored secret into the environment still exposes it to agent-generated code” [7].

NVIDIA and OpenAI document the same fix, a broker outside the sandbox. OpenShell’s README: “Agents never see real credentials; OpenShell adds them only to requests bound for approved endpoints” [8]. OpenAI’s docs, for its hosted sandboxes: “Sandbox code uses a placeholder; a network proxy supplies the real secret for approved hosts.” For self-hosted setups they say to run a trusted proxy or server outside the environment, and for function tools to keep credentials in the application that handles the call and return only the result [7]. Code in the sandbox never sees the secret, and a placeholder sent to any other host carries nothing.

Where an agent must hold a token, keep it narrow and short-lived. GitHub’s installation access tokens, for example, can be limited to named repositories and permissions, and expire after one hour [9]. Keep a list of every credential each agent can reach, because revocation is the cleanup: after the token incident OpenAI deactivated all the users’ keys, and later every employee’s key as a precaution [2].

Confine the sandbox and keep the record outside it

OpenAI’s guidance is to run agent workloads “in isolated compute, such as virtual machines”, with separate environments for users or workloads that must not share data [7]. OpenShell runs each agent in its own sandbox, where “kernel controls confine which files it can access and which system calls it can make” [8]. On 25 September GitHub put local sandboxing for the Copilot app into public preview, to limit agents’ access to files, networks and credentials [10]. Mount only what the task needs, read-only where you can.

The agent’s own record needs the same protection. A preprint from 24 September ran local coding-agent harnesses in containers with full access, and all but one let the agent delete its own trace when asked [11]. The authors recommend a “trusted interception server, outside the agent host’s control” that writes every model request and response to an append-only log before forwarding it, and fails closed if recording fails. They note that such a log cannot show whether a tool call actually ran as the trace reports [11].

So we also log at the enforcement points: the egress proxy, the credential broker and the approval service sit outside the sandbox and see every request that crosses the boundary as it happens, and their logs go to storage the agent holds no credentials for. Someone also has to watch them. In OpenAI’s case the DNS activity was logged, but an infrastructure detector for anomalous DNS activity excluded the affected environment; OpenAI is now deploying more DNS detections and tests of the detection pipeline [1].

Hard limits and an automatic stop

OpenAI’s monitor did its part and the run still went on: “the run did not stop automatically as expected, leading to confusion around whether it should have been stopped” [1]. OpenAI says it is addressing these operational gaps. After the token incident, it brought that model back with an asynchronous monitor on its chain of thought (CoT) “which could take down the model within minutes of a misaligned CoT” [2]. NVIDIA’s Sentry, a reference design, runs its watchdog out of band on BlueField-4 DPUs, and NVIDIA says it can quarantine an agent that tries to move outside its boundaries “in milliseconds” [4]. Decide in advance which alerts end a run, and wire them to do it: stop the process, revoke the run’s credentials, keep the sandbox for inspection, and leave the restart to a named person.

Caps hold whether or not anything is detected. The OpenAI Agents SDK raises a MaxTurnsExceeded exception when a run passes its max_turns limit [12]. Monitors can add a cap of their own: the production monitors in the preprint end a session after a set number of blocked actions [3]. Spend needs a hard cap too: on OpenAI’s platform, “Spend alerts do not enforce a cap”, while a monthly hard limit makes affected requests fail with a 429 error, though enforcement is not instantaneous [13]. A project limit covers only that project’s traffic, so an agent with its own project gets its own cap [13]. OWASP adds rate limits on agent actions, which cap how many bad actions fit in a given period and give monitoring time to catch them [14].

A person approves what cannot be undone

Some actions cannot be rolled back: a payment, a deletion, an email to a customer, a push to a public repository. OWASP’s guidance is to “require a human to approve high-impact actions before they are taken”, and to “implement authorization in downstream systems rather than relying on an LLM to decide if an action is allowed or not” [14].

OpenAI’s computer-use docs note that “Asking for confirmation through a function tool relies on the agent calling that function.” To guarantee confirmation before consequential actions, they say to “restrict the hosted browser to resources that cannot perform them, or use a browser runtime you control” [15]. In the OpenAI Agents SDK the approval rule sits on the tool: the runner checks it when the model emits a call, and pauses the run with the tool name and arguments until a person decides. Approval for local shell and file-patch tools is off unless you turn it on [16]. This holds only while the runner sits outside the sandbox; NVIDIA’s release describes Anthropic’s Claude Managed Agents running the agent loop “in a separate server from the sandboxes where their work executes” [4].

Approve the exact action, with nothing editable between approval and execution. The preprint found that production monitors let edits inside the working directory through unreviewed, so an approved nohup sleep 30; ./run.sh & could run a run.sh rewritten after the approval [3]. Automatic approval helps with volume: GitHub’s assisted approvals for Copilot agent sessions in JetBrains, in public preview, approve low-risk tool calls and prompt for higher-risk ones [10]. Keep a fixed list of irreversible actions that always go to a person, whatever an automatic approver decides.

Checklist

Keep your controls written down, dated and owned, and test them again whenever the agent’s tools or environment change.

From inside the sandbox, a direct HTTPS request to an unlisted host, a query to a public DNS resolver and a lookup of an unlisted domain all fail.
With either egress layer switched off in a test environment, the traffic is still blocked.
No live secret turns up in the sandbox’s environment variables, files or mounts, and authenticated calls work only through the broker, to approved hosts.
Any token the agent holds is scoped to its task and expires within hours, and each agent has a list of the credentials to revoke.
The agent cannot write to its logs, its policies or code an approved command will run; logs come from the interception server, proxy, broker and approval service.
Passing a turn, tool-call, token or time cap ends the run, and the provider’s monthly spend limit blocks requests instead of only alerting.
A synthetic high-severity alert stops the run and revokes its credentials in a time you have measured, and a named person owns the restart.
Every action on the irreversible list pauses for a person, enforced in the tool or downstream system, with the exact arguments shown.

This is the order we work through when we build agents and automation for production.

Sources

This post first appeared on Dravin AI.

The 90 firms that fix AI-generated code

info@dravin.ai (Dravin AI) — Mon, 21 Sep 2026 18:30:00 GMT

TL;DR

We found 90 firms that fix AI-built apps; our descriptions of 48 of them mention security.
Published timelines run from a 48-hour security sweep to re-architecture over six to twelve weeks.
A common sequence: read the code, stabilise what is dangerous, then refactor or rebuild what will not hold.

In September 2026 we went looking for companies that sell cleanup of AI-generated code: apps built with Lovable, Bolt, Cursor, Replit, v0, Claude Code and similar tools, by people who then got stuck. We found 90. This post is what those 90 say they fix, how they package the work, and what we think a buyer should demand from any of them.

Why this category exists

Our one-line descriptions of the 90 firms mention the kinds of situation that lead to a cleanup: a business owner, a product manager or a COO builds something real with an AI tool, then meets real users, a payment that fails, or a security email from a stranger.

The demand is visible in public. On r/ExperiencedDevs, the thread “Getting more calls to fix ai generated codebases than actual new builds lately” had 405 upvotes and 102 comments when we read it in September 2026; one comment: “Cleanup contracts are gonna be a whole market segment.” On r/SaaS, an MVP dev shop said it lost half its pipeline to Claude Code in 2025; of the prospects who tried it instead, about a third shipped, a third broke in production, and three came back for cleanups.

In Veracode’s July 2025 study, more than 100 LLMs completed 80 coding tasks with a known potential for security flaws, and in 45% of test cases they introduced a vulnerability from the OWASP Top 10.

45% of test cases where code from more than 100 LLMs introduced an OWASP Top 10 vulnerability, on 80 tasks with a known potential for security flaws. Source: Veracode, July 2025

Who the 90 are

Type of firm	Count
Established dev agencies that added a cleanup service line	48
Human audit specialists	13
Agencies formed for rescue work	10
Automated scanners	10
Solo fixers and freelance studios	6
Marketplaces and expert networks	3

Twenty are in the USA, 18 in Europe outside the UK, 7 in India, 6 in the UK, 4 in Canada, 11 elsewhere, and 24 give no location. Size is mostly undisclosed (53 of 90); of the rest, 6 have 250 or more people, 10 are mid-sized and 21 are small, micro or one-person.

We could open 75 of the 90 websites; 15 are known only from directory listings, and three firms’ sites no longer load. Our descriptions of 55 of the firms name an AI builder; Lovable leads (50), then Cursor (46), Bolt (44) and Replit (41).

What they say breaks

We counted which failure classes our one-line description of each firm names. A one-line description leaves detail out, so a firm’s own pages can cover more than these counts show.

Failure class in our one-line description of each firm	Entries (of 90, scanners included)
Security in general (vulnerabilities, OWASP, hardening)	48
Missing or broken tests	25
Performance and scalability	25
No deploy pipeline, CI/CD or hosting setup	23
Authentication and access control (including Supabase RLS)	18
Data model or database problems	15
Secrets and API keys in code	10
Monitoring and observability	9
Documentation	9
Payments and Stripe	8
Hallucinated APIs	1

Security is the most common theme. Our descriptions of 48 of the 90 mention it, and seven of the ten automated scanners are security scanners: Supabase row-level-security probes, leaked-key detection, public buckets. Sidekick Interactive’s case study is typical of the genre: exposed API keys, database holes and broken payments in one app.

Structure carries a different risk: duplicated logic and unsafe coupling make an AI-generated codebase slow and risky to change. A security patch is usually local; untangling several copies of the same auth check, each slightly different, is where the work goes.

Spend is worth a question too. On r/cursor, a team described a Cursor agent that a PM had asked to tag 87 tasks: it looped for an hour and ran up a bill far above the team’s usual daily spend. That was a coding agent’s bill, not an app’s, but an app that calls a model on every request with no budget, rate limit or per-user cap is a bill waiting to happen in the same way.

How they package it

The 80 human-service firms (the 90 minus the 10 scanners) describe their offer in a few recurring ways, often more than one at once.

Packaging pattern	Firms (of 80)
Audit, assessment or review mentioned	61
Audit or assessment promised within 48 hours	9
Work described in sprints	9
Rebuild or re-architecture offered as one option	9
Explicitly against rebuilding	5
Retainer, maintenance or monitoring mentioned	16

Audit first. Valletta Software (Malta) is the cleanest version: a senior engineer reviews the repository across eight areas, delivers the audit in three to seven business days, depending on the tier, with a debrief call, then scopes the cleanup, typically six to eight weeks. MetaCTO runs a 48-hour code audit, then promises production-ready in 30 days, with weekly demos.

Fixed-scope rescue. AxonBuild sells a 10-working-day production-engineering sprint covering auth, secrets, payments, CI/CD and monitoring, ending in a verified handover with two weeks of defect cover and 30 days of async questions about the handover and architecture. Relux Works (Armenia) runs a one-week audit, then a two-to-three-week stabilisation sprint, then what it calls “de-vibe-coding”: moving the app to a production architecture.

Rebuild. Nine firms offer a rebuild or re-architecture as one option, and five say outright that they avoid a full rewrite. Bitnoise (Poznań) shows the whole sequence without one: audit in three to seven business days, a stabilisation sprint of two to four weeks, then re-architecture over six to twelve weeks, which it says is not a from-scratch rewrite.

Retainer. Vibe-Audit (Barcelona, run by one engineer) runs a pre-launch security sweep delivered in 48 hours and done-for-you fixes, then offers past clients a review of new releases, up to a set number.

Side by side, a common sequence emerges: read the code, stabilise what is dangerous, then refactor or rebuild what will not hold. Watching it afterwards comes up less often: 16 of the 80 mention a retainer, maintenance or monitoring.

How cleanup is sold

Read the code: audit or review first
Stabilise: what is dangerous now
Refactor or rebuild: what will not hold
Watch it: retainer or monitoring

A common sequence among the 80 human-service firms: 61 mention an audit or review, and 16 mention a retainer or monitoring.

What a good cleanup engagement includes

Use this on anyone you are considering.

A written review before any commitment, with findings ranked by severity and a verdict: ship, patch, refactor or rewrite. If the verdict is “not worth fixing”, they should say so.
An experienced engineer who has actually read the code. A scanner finds leaked keys; it does not find the auth check that is bypassed on one route.
Security tested against the running app, not only the repository: authentication, per-row authorisation, secrets rotated rather than just deleted from git, public storage buckets closed.
Characterisation tests written before refactoring, so current behaviour is pinned and the cleanup cannot silently change it.
Duplicated logic collapsed and the data model reviewed, because that decides whether the next feature takes a day or a month.
A deploy pipeline and monitoring at the end: staging, CI, error tracking, backups. Working on one person’s laptop does not count as done.
AI API spend bounded: budgets, rate limits, per-user caps and an alert. Ask for it by name.
Handover: documentation, and you holding the repository, the cloud accounts and the keys. You own the code from day one.
A fixed scope with a definition of done and a defect-cover window afterwards.

How we do it

Dravin AI is an AI software company. We build AI systems and software, and we clean up AI-built apps. Against the list above, this is what we commit to:

A written report first. Read-only access to one repository, and within 48 hours a written report with findings ranked by severity: the file, the risk and the fix, and what to leave alone. If nothing needs fixing, it says so. There is no obligation to go further.
Secrets. If we find a secret committed to your repository, we tell you the same day, ahead of the report, and help you rotate it. Where we hold the access, we rotate it with you within 24 hours.
Auth and data access. Every route’s auth check, database access policies included, is traced in the code. Testing the running app is agreed on the call, because it needs access beyond the repository.
Tests before refactoring. Tests on the paths that make money, run on every pull request, and structure changed in small, tested steps.
Deploys. From the repository, with a rollback that has been tried.
AI API spend. Timeouts, budgets and retry caps on every model call.
Ownership. The code is yours from the first commit, we work on branches, never main, and our access is removed within seven days of handover, confirmed in writing.
Defect cover. Anything that misses the statement of work, reported in writing within 30 days of handover, is corrected under it.

An engagement starts with a call about what you built, what it does in production and what worries you. If that is an app that got you to traction, book one or request a code review.

Method

Every count comes from the cleanup list in our research workbook: 90 firms, each with a segment, a region, a size band, a one-line description of what it does, its delivery model and its stated turnaround. We read the list on 22 September 2026. The workbook itself is not published; these are the counting rules.

Type, region, size: counts of each firm’s segment, region and size band. “Small, micro or one-person” is Small (10 to 49) plus Micro (2 to 9) plus Solo.
Reachability: whether we could fetch the firm’s site or know it only from a directory listing, and whether its site still loads, meaning its domain still resolves.
Tool mentions: a case-insensitive search of each firm’s description for each builder’s name.
Failure classes: a case-insensitive keyword search of each firm’s one-line description only. Security: secur, vuln, OWASP, pentest. Tests: test, QA; Sherlock Forensics matches only through “pentest” and is excluded, so 25 rather than the mechanical 26. Performance: perf, scalab, bottleneck. Deploy: CI/CD, CI, deploy, pipeline, hosting, DevOps. Auth: auth, access control, RLS. Data model: data model, database, DB, schema, persistence, RLS. Secrets: secret, key(s), credential. Monitoring: monitor, observab, Sentry. Documentation: doc, docs, document or documentation as whole words. Payments: Stripe, payment. Hallucination: hallucinat. Scanner types: the descriptions of the 10 automated scanners.
Packaging: over the 80 firms that are not automated scanners, searching the description, the delivery model and the stated turnaround unless a rule says otherwise. Audit or review: audit, assess, review, scan, diagnos or scorecard. 48 hours: the turnaround promises an audit, assessment, scan or check result in 48 hours or less; fix times and reply times excluded. The nine: MetaCTO, Pfaff Digital, Sonder, VibeAudits, Mitrix Technology, ShipClarity, Vibe-Audit, VibeCodeBlue and one solo fixer. Sprints: “sprint” anywhere in those three fields. Rebuild offered: rebuild, re-architect, rewrite or full build (11 firms), minus the two of them that also say “without rebuild”, “no rewrites”, “not rebuild”, “diagnosis-over-rebuild” or “without starting from scratch”. Against rebuilding: one of those five phrases in those three fields, the firm’s hero line or the workbook’s notes on how each firm positions itself (5 firms). Retainer: retainer, maintenance, ongoing, monitoring, subscription, defect cover or post-rescue support (16 firms).
Named offers, turnarounds and claims are quoted from what each firm publishes, unaudited.
Reddit and studies: r/ExperiencedDevs (405 upvotes, 102 comments), r/SaaS (65 upvotes, 66 comments) and r/cursor (228 upvotes) from our AI-agency landscape research (48 searches, 2,837 posts, 17 September 2026); Veracode’s 45% from its July 2025 press release. The links are under Sources.

Sources

r/ExperiencedDevs, “Getting more calls to fix ai generated codebases than actual new builds lately”, posted 22 April 2026, read 17 September 2026. reddit.com
r/SaaS, “I lost half my agency’s pipeline to Claude Code in 2025. Here’s the honest take on who should use it instead of paying me”, posted 21 April 2026, read 17 September 2026. reddit.com
Veracode, “AI-Generated Code Poses Major Security Risks in Nearly Half of All Development Tasks, Veracode Research Reveals”, press release on its 2025 GenAI Code Security Report, 30 July 2025. veracode.com
Sidekick Interactive, “Finish & Launch Your Vibe Coded App”, with the case study. sidekickinteractive.com
r/cursor, a team’s account of a Cursor agent that looped while tagging 87 tasks, posted 10 June 2026, read 17 September 2026. reddit.com
Valletta Software, its guide to how a vibe coding audit works, with turnaround by tier, updated June 2026 and checked 6 October 2026. vallettasoftware.com

This post first appeared on Dravin AI.