One of the most discussed studies of 2025 came from METR, a research nonprofit. Their paper, *Measuring the Impact of Early-2025 AI on Experienced Open-Source Developer Productivity*, produced a counterintuitive result worth understanding carefully.
(Our plain-language summary; the paper is linked so you can read the source.)
How the study worked
This wasn't a survey — it was a randomised controlled trial, the gold standard. Experienced open-source developers worked on real tasks in codebases they already knew well. For each task, AI assistance was randomly allowed or not allowed, and the time taken was measured.
The surprising result
When allowed to use AI tools, developers took about 19% longer to finish their tasks. And the twist: those same developers believed the AI had made them roughly 20% faster. A nearly 40-point gap between perception and reality.
Why the slowdown — and why it's nuanced
The result is narrow and easy to misread. It applies to experts working in code they know deeply — exactly the situation where AI suggestions add review and correction overhead rather than saving lookup time. The likely causes:
- Time spent reading and verifying AI output.
- Fixing confidently-wrong suggestions.
- Context-switching between writing and prompting.
It does not mean AI makes everyone slower. For unfamiliar code, boilerplate, or less-experienced developers, the picture can be very different.
The headline isn't "AI is useless." It's "people are bad at judging their own productivity" — and that perception gap is dangerous when you're deciding where to invest.
What teams should take from it
- Measure, don't assume. Feeling faster isn't the same as being faster — instrument real outcomes. (This is the heart of measuring AI ROI.)
- Match the tool to the task. AI helps most where the human lacks context, not where they have the most.
- Beware vibes-based rollouts. Enthusiasm is not evidence.