3 SEP 2026 — Software engineering research has spent three years bolting scaffolding onto large language models. A University of Virginia team asked what survives when a newer model arrives. They reproduced 35 techniques from the ICSE 2026 research track, the field's main conference, and tested each against a single automatically generated prompt to that newer model. Between 37 and 63 per cent were outperformed, and the spread is not sloppiness: it reflects how mixed results are scored, and the authors say their sample is too small for a significance test.
What the study did
The paper is "What Survives the Next Model? Benchmarking LLM-Based Techniques Against Single-Prompts", posted on 31 August by Nahian Salsabil, Joy Saha, Simantika Bhattacharjee Dristi, Nicholas Phair, Nusrat Jahan Mozumder, Matthew B. Dwyer and Sebastian Elbaum.
They began with 321 ICSE 2026 papers, filtered to 47 eligible ones and evaluated 35, reproducing each technique's core result. The original work used a spread of models including Claude 3.5 Sonnet, GPT-4o, Llama, DeepSeek, Qwen and Gemini variants. The comparison used Claude Sonnet 4.6, released in February 2026, prompted once.
The tasks span most of what the field has been applying these models to: code generation, bug finding, program repair, impact analysis, requirement formalisation, log analysis, test generation and verification.
Disclosure: RECATOOLS is written with the assistance of Claude, made by Anthropic, whose model is the newer one in this comparison.
Why the finding is a range
The study sorts each comparison into outperforming, mixed or underperforming. Count the mixed cases as wins and you get one figure; count them as losses and you get another. The authors report the interval in between.
The authors list that coarseness themselves as a threat to construct validity, alongside three others: the sample comes from one venue, the reproduction was partial across 35 distinct research groups working differently, and budget limits capped how deeply some papers could be evaluated. On conclusion validity they state plainly that the number of papers is too small to test for significance.
The result does not mean half of software engineering research was wasted. It means that on a single venue's worth of work, a substantial and imprecisely bounded fraction of the engineering around the model stopped paying for itself within about a year.
Single prompt does not mean naive prompt
The baseline was generated automatically, not written and tuned by hand. That cuts both ways.
In the authors' favour, the baseline was not hand-optimised to win, which would be the obvious way to rig the experiment. Against the simplicity story, the thing that beat the pipelines was itself generated by a system, so "one prompt beats your pipeline" is less plain than it sounds.
We tested the adjacent claim in August, running 474 answers against the prompting folklore, and found the popular tricks made no measurable difference. This paper is the other half of that result. The value was assumed to be in the scaffolding around the prompt, and much of that scaffolding did not survive one model generation either.
Why this keeps happening
Retrieval, decomposition, verification, repeated sampling: each of these compensates for something the model of the day could not do by itself. A year later the model can, and what is left is overhead plus a paper that documents a limitation somebody else removed.
That does not make the research a failure. It is what happens when the substrate improves faster than the publication cycle: a conference paper written against a model available eighteen months earlier is describing a system that no longer exists.
The uncomfortable part is that the field's incentives do not notice. A paper is accepted, cited and counted on the strength of a comparison against the model of its moment, and nothing in the process re-runs it afterwards. This study is unusual precisely because somebody did.
What the authors actually recommend
Their conclusion matters more than the headline number. They argue the community should stop chasing model capability and build techniques that scale alongside a better model rather than being erased by one.
They put symbolic methods in that category: approaches with guarantees a language model does not provide, which a stronger model makes more useful rather than redundant. A verifier that proves a property does not become unnecessary when the generator improves; it becomes cheaper to satisfy.
The distinction they are drawing is between scaffolding and complementing. Scaffolding comes down once the building stands on its own. Complementing means doing a job the model was never going to do, which is why a stronger model makes the complement more useful rather than redundant.
What this means for anyone building on models
For people building on models outside research, the lesson is direct. An elaborate pipeline is a bet on a capability gap persisting, and this paper puts a recent number on how long such gaps have lasted, which is about twelve months.
So keep the scaffolding separable and re-test the plain baseline when the model changes. Almost nobody does, because the pipeline works and re-testing costs money. None of this argues for abandoning engineering: 37 to 63 per cent were outperformed, which leaves a substantial remainder that was not.
Two cautions are in order. The paper is an unreviewed preprint, and its general claim about how fast scaffolding decays rests on a single year's work from one conference tested against one newer model.
View author profile → · Editorial policy
