It's become an article of faith that AI coding tools make developers dramatically faster. Then METR ran a proper randomised controlled trial — and the result surprised everyone, including the developers in it.
The study
METR's July 2025 paper, *Measuring the Impact of Early-2025 AI on Experienced Open-Source Developer Productivity*, followed 16 experienced developers through 246 real tasks on repositories they maintain (~1M lines of code). Tasks were randomly assigned AI / no-AI — the same design used in drug trials.
The result
- Allowing AI increased completion time by 19%.
- Developers believed AI made them 20% faster — a 39-point perception gap.
- Economists and ML experts had predicted a ~38–39% speed-up. Everyone was wrong in the same direction.
AI slowed people down while making them feel faster. Perception is not a measure of productivity.
This is one study, on experts in code they know deeply — the case where AI helps least. It doesn't prove AI never helps. But if you justify AI tooling on productivity, measure it.
Sources
Written by ivector
Start a project →