One of the most discussed studies of 2025 came from METR, a research nonprofit that studies AI capabilities. Their paper, Measuring the Impact of Early-2025 AI on Experienced Open-Source Developer Productivity, produced a counterintuitive result worth understanding carefully, because it's easy to weaponise in either direction.
(Our plain-language summary; the paper is linked so you can read the source.)
The backdrop
By early 2025, the prevailing story about AI coding assistants was one of obvious, large productivity gains. Survey after survey reported developers feeling dramatically faster, and vendor case studies quoted big percentage improvements. But almost all of that evidence was self-reported (people saying how much faster they felt) rather than measured. METR set out to do something the hype cycle had mostly skipped: actually time the work.
How the study worked
Crucially, this wasn't another survey. It was a randomised controlled trial, the same gold-standard design used to test medicines. Experienced open-source developers worked on real, substantial tasks in codebases they already knew deeply, often projects they personally maintained. For each task, AI assistance was randomly allowed or not allowed, and the actual time taken to complete the work was recorded. Randomisation is what makes the result credible: it isn't comparing different people or cherry-picked tasks, it's the same skilled developers doing comparable work with and without the tools.
The details matter for how far the result travels. METR recruited 16 developers who had spent about five years, on average, on the projects they worked in. Between them they brought 246 real issues from their own repositories, and the tasks averaged about two hours each. When AI was allowed, they mostly used Cursor Pro with Claude 3.5 and 3.7 Sonnet, which were frontier models in early 2025. So this is a study of veterans on home turf with the best tools of that spring, not of a typical team.
Why does that design matter so much? Because the usual way these claims get made is hopelessly biased. If you ask people whether a tool helped, they remember the moments it dazzled them and forget the quiet minutes spent untangling its mistakes. A controlled trial sidesteps memory and impression entirely; it just measures the clock. That's also what makes the result hard to wave away: there's no "but they weren't using it properly" escape hatch, because these were skilled developers using current tools on their own code.
The surprising result
When allowed to use AI tools, developers took about 19% longer to finish their tasks. Read that twice: longer, not shorter. And here's the twist that gave the study its punch: those same developers believed the AI had made them roughly 20% faster. Even after the fact, having lived through both conditions, they misjudged the direction of the effect. That's a nearly 40-point gap between what people felt and what actually happened.
Here is every forecast in the paper's abstract next to the measurement:
| Who | Expected effect on task time |
|---|---|
| The developers, before starting | 24% less time |
| Economics experts | 39% less time |
| Machine-learning experts | 38% less time |
| The developers, after finishing | 20% less time |
| Measured in the trial | 19% more time |
Every forecast got the direction wrong, including the developers' own estimate after they had done the work.
Why the slowdown, and why it's genuinely nuanced
It would be a serious mistake to read this as "AI makes developers slower, full stop." The result is narrow and specific. It applies to experts working in code they know intimately, precisely the situation where a human already holds most of the context in their head, so AI suggestions add review and correction overhead instead of saving lookup time. The likely contributors:
- Time spent reading and verifying AI output before trusting it.
- Fixing suggestions that were confidently wrong or subtly off.
- Context-switching between writing code and steering the assistant.
- Over-reliance on prompting for things a fluent expert would simply type faster by hand.
Flip the conditions and the picture can reverse. For unfamiliar codebases, boilerplate, unfamiliar languages, or less-experienced developers who lack that internal context, the same tools can deliver real speed-ups. The study measured one demanding scenario well; it did not measure all of them.
The headline isn't "AI is useless." It's "people are remarkably bad at judging their own productivity," and that perception gap is dangerous precisely when you're deciding where to spend money.
What METR found when it ran the study again
METR didn't leave the 19% standing as a permanent fact. On 24 February 2026 it published early results from a continuation on late-2025 tools: 57 experienced open-source developers, 143 repositories and more than 800 tasks, again randomised task by task. The central estimates now point the other way. Developers from the original study finished tasks in about 18% less time with AI, with a confidence interval running from 38% less time to 9% more. Newly recruited developers took about 4% less time, with an interval from 15% less to 9% more.
Don't swap one headline for the other, though. METR calls its own new data "only very weak evidence" and says the central estimate is "likely a bad proxy for the real productivity impact of AI tools on these developers." The reason is instructive. Between 30% and 50% of developers told METR they were choosing not to submit some tasks, because they didn't want to do them without AI. The tasks people now refuse to do by hand are the ones most likely to show a speed-up, so they drop out of the measurement. METR said it is changing the experiment design because of this.
Then, on 11 May 2026, METR published a survey of 349 technical workers who self-reported AI speed gains of up to 3x, and in the same post it flagged that the most extreme answers looked overstated. We read that survey in a separate piece. Our read of all three: the 2025 trial is now a snapshot of early-2025 tools, not a current number. Its main point has held up better than its headline, because the felt speed-up and the measured one keep coming apart.
What teams should take from it
- Measure, don't assume. Feeling faster is not the same as being faster, so instrument real outcomes like cycle time, throughput and rework rates. This perception-versus-reality gap is the entire reason measuring AI ROI properly matters, and it's the same result our companion piece on does AI actually make developers faster unpacks from the other side.
- Match the tool to the task. AI tends to help most where the human lacks context, and least where they already have the most. Roll it out where the gap is real.
- Beware vibes-based rollouts. Genuine enthusiasm from your team is wonderful, but it is not evidence. Run a small controlled comparison, the kind an evaluation harness makes repeatable, before you commit budget across the org.
There's also a leadership lesson tucked inside the perception gap. If your own engineers (the people closest to the work) can misjudge the effect of a tool by nearly forty points, then dashboards built on self-reported satisfaction or anecdotal "this saved me so much time" feedback are a shaky basis for a six- or seven-figure rollout decision. The fix isn't to distrust your team; it's to give them an honest measurement instead of asking them to estimate. Pick a couple of representative task types, run them with and without the tooling for a few weeks, and look at the clock and the rework rate rather than the mood in the room.
The deeper point is that this study is a model for how to evaluate any AI initiative: a quiet, measured trial beats a confident anecdote every time. If you'd like help measuring your own team's AI-assisted output with an honest before-and-after design rather than guessing, we're happy to help set one up. And looking forward, expect the answer to keep shifting as tools and workflows mature, which is exactly why the habit of measuring, not assuming, is the durable takeaway.
Sources
- METR (2025): Measuring the Impact of Early-2025 AI on Experienced Open-Source Developer Productivity
- METR (10 July 2025): Study page with the tools, task lengths and caveats
- METR (24 February 2026): We are changing our developer productivity experiment design
- METR (11 May 2026): Measuring the self-reported impact of early-2026 AI on technical worker productivity
