TORONTO, 9 AUG 2026 — Researchers at 1Password's Off-by-1 Labs put two frontier models to work fixing real security vulnerabilities. About a quarter of the patches cleanly fixed the flaw. The rest either failed to remediate it or broke something else.

The headline finding is the success rate. The finding with consequences is what the failures looked like, because several of them pass your test suite.

What was tested

The lab set two current models, ChatGPT 5.5 and Claude Opus 4.8, to generating security patches with a serious budget — the exact task they are most often proposed for.

Roughly one patch in four fixed the vulnerability cleanly. The remainder divided into three recognisable failure modes, and the ordering matters more than the ratio.

Two numbers are circulating and they are not measuring the same thing. Reporting of the work puts clean fixes at about a quarter, while a separate account gives a 47% success rate against 53% that failed to fully remediate without introducing new problems. The gap is a definitional one — a patch can reduce a vulnerability without cleanly closing it — and we quote both rather than pick the flattering one.

The sample is the caveat that matters more than either figure: six high-complexity CVEs. That is a small enough set that the percentages should be read as an indication of failure modes rather than a rate you could plan capacity against.

Some patches addressed only a subset of the vulnerable code paths, leaving the flaw reachable. Others added fragile guard code that satisfied the tests without addressing the root cause. A third group introduced subtle changes to application behaviour while closing the immediate hole.

Why the second failure mode is the dangerous one

A patch that obviously does not work is cheap. It fails a test, or a reviewer reads it and rejects it, and the cost is a few minutes.

A patch that adds a guard which satisfies the tests while leaving the root cause intact is expensive in a specific way: every automated signal you have says the vulnerability is fixed. The test suite is green. The pull request is approved. The ticket closes. The finding disappears from the backlog, and the only thing that has actually changed is that nobody will look at it again.

That is worse than an unpatched vulnerability, because an unpatched vulnerability is still on somebody's list. This class of failure converts an open item into a closed one without changing the underlying exposure, and closed items do not get re-examined until an incident.

The third mode — subtle behaviour changes made while fixing something else — is the one that shows up months later as a bug nobody can trace, in code whose git blame points at a security fix everyone agreed was necessary.

The reviewing cost is the real result

The researchers make a point that deserves more attention than the percentage, and it is an economic one rather than a technical one.

Reviewing a large volume of mostly-incorrect, similar-yet-subtly-different generated patches imposes a cognitive load that will likely exceed the effort of understanding and patching the vulnerabilities directly.

That reframes the whole proposition. A tool with a 25% success rate is still valuable if verifying its output is cheap — the arithmetic works out fine when checking takes a fraction of the time doing would. It is negative value if verification costs more than the original work, and security patches are close to the worst case for verification: to know whether a patch actually closes a vulnerability, you have to understand the vulnerability, which is most of the work of fixing it yourself.

Four patches that look plausible and differ in ways that matter is a harder review task than one blank editor.

A second study, a larger sample, the same direction

Veracode's 2026 GenAI Code Security Report ran a broader test — 100 frontier models — and found something adjacent and worse: 44% of the code produced introduced a detectable OWASP Top 10 vulnerability, with average pass rates around 56%.

That is code generation rather than patching, so it is not the same measurement. It points the same way, and from a sample large enough to carry more weight than six CVEs: models write code that works more often than they write code that is safe, and the gap between those two is where security debt accumulates.

Tim Jarrett, Veracode's vice-president of product, draws the operational conclusion plainly: it is "premature to treat those as anything other than another code change... that needs to be reviewed" rather than merged autonomously. That is a modest claim and the studies support it; the immodest claim, that generated patches can be trusted on test evidence, is the one with no support behind it.

The same failure, measured a different way

The pattern is now appearing from two independent directions.

A permission-approval study measured how well humans catch dangerous AI agent commands and found they miss a third of them — with the misses concentrated on requests that look routine: npm run analyze was waved through 64.7% of the time. Here, engineers are asked to catch subtly-wrong patches among plausible ones.

Both are the same task. Both studies put a human at the end of a pipeline and ask them to spot the one bad output in a high-volume stream of plausible-looking ones. Humans are measurably poor at that, and the poorness scales with volume.

The industry, in other words, is building an oversight model based on the one task people are demonstrably bad at.

What this does not say

It does not say models are useless at security work, and the study's shape sets the limits on what it does say.

Two models were tested, at particular effort settings, on a particular set of vulnerabilities, autonomously. "Autonomously" is doing significant work in that sentence. A model guided by a reviewer mid-task is a different system from one handed a flaw and asked to produce a diff. Most competent practice already looks like the former.

Models are also demonstrably good at neighbouring tasks like summarising unfamiliar code, drafting a regression test, or proposing candidate locations for a fix. None of those requires the output to be correct without review, because the human is reading the code anyway.

The finding is narrower and firmer: autonomous patch generation, accepted on the strength of a green test suite, does not currently work.

What to do with this

Do not let a generated security patch merge on test evidence alone. The failure mode most likely to reach production is precisely the one your tests approve, so tests cannot be the gate for this class of change.

Require the reviewer to state the root cause in their own words before approving. It is a small ceremony and it directly targets the guard-code failure — you cannot confirm that a patch addresses a root cause you have not articulated.

Use the models where verification is cheap. Finding candidate vulnerable paths, explaining unfamiliar code, and drafting the test that proves a fix works are all high-value, trivially checkable tasks for a model. The task to be careful with is the one where the output is a diff nobody wants to read closely.

And count the review time honestly. If your team is spending longer assessing generated patches than it would spend writing them, the tool is not accelerating anything, and the only way to know is to measure it rather than assume it.

What to watch

Will anyone test a human-in-the-loop workflow, which is how most teams actually work? Will the success rate jump with the next model generation? And will any vendor selling autonomous patching publish its own remediation rate? They have the data.