Skip to content
← Back to blog
Research·Updated October 22, 2026·7 min read

Does AI actually make developers faster? What the evidence says

In a 2025 randomised trial, experienced developers took 19% longer with AI tools while believing they were 20% faster. Newer evidence complicates the story.

It's become an article of faith that AI coding tools make developers dramatically faster. Then METR ran a proper randomised controlled trial, and the result surprised everyone, including the developers in it.

The study

METR's July 2025 paper, Measuring the Impact of Early-2025 AI on Experienced Open-Source Developer Productivity, followed 16 experienced developers through 246 real tasks on mature repositories they had contributed to for years (five years each on average, in codebases of over a million lines). Tasks were randomly assigned AI / no-AI, the same design used in drug trials.

The design matters because it's the part most "AI made us X% faster" claims skip. These weren't toy problems or unfamiliar codebases; they were real issues in repositories the developers knew intimately. Randomly assigning AI or no-AI to each task removes the usual confound (that people reach for AI on the easy tasks and grind through the hard ones by hand) and lets you actually attribute the difference to the tool.

The result

  • Allowing AI increased completion time by 19% (METR later gave the confidence interval as +2% to +39%).
  • Before starting, developers forecast that AI would cut their time by 24%. Afterwards they still believed it had cut it by 20%, when it had raised it by 19%.
  • Economists and ML experts had predicted tasks would take 38 to 39% less time. Everyone was wrong in the same direction.
AI slowed people down while making them feel faster. Perception is not a measure of productivity.

Why a slowdown, and why it felt fast

METR does not name a single cause. It lists contributing factors, among them developers' deep familiarity with their repositories, the size and age of those codebases, and the low reliability of the AI output: developers accepted fewer than 44% of AI generations. One plausible reading makes the result less paradoxical than it sounds. On code you know deeply, the bottleneck isn't typing; it's understanding. The AI produces plausible code quickly, but then you have to read it, check it against context the model didn't have, and fix the parts that are subtly wrong. That review-and-repair loop can cost more than just writing the change yourself. Meanwhile it feels faster because the screen fills with code almost instantly; the effort moves from "producing" to "verifying," and verifying is quieter work that doesn't register as labour the same way. That's the perception gap in a sentence: output volume is not the same as progress.

A second reading is more speculative. AI lowers the activation energy of starting, which feels great: a blank function gets a body in seconds. But on mature code the hard part was never starting; it was getting the last 20% exactly right against constraints the codebase already encodes. The model is fast at the easy part and unreliable at the hard part, so it front-loads the satisfying work and back-loads the expensive work. You end the task tired from reviewing rather than writing, and your memory of "how it went" is dominated by that brisk, productive-feeling start.

It's worth being precise about scope. This is one study, on experts working in code they know deeply, close to the worst case for AI assistance. It does not show AI never helps. For unfamiliar languages, boilerplate, exploratory prototyping or developers who are new to a codebase, the picture may look very different. Earlier controlled studies on simpler or enterprise tasks found speed-ups: developers with GitHub Copilot finished a set task (an HTTP server in JavaScript) 55.8% faster, and a Google trial with 96 engineers put the gain at about 21%, with a large confidence interval. Results depend heavily on setting. What METR's trial punctures is the assumption that the speed-up is automatic, large and self-evident.

METR has also revisited its own result. In February 2026 it said a follow-up experiment was too distorted by selection effects to give a reliable number, and that developers are likely more sped up by AI tools now than in early 2025. Read the 2025 result as a snapshot of early-2025 tools, not a permanent verdict.

It also lands inside a broader pattern. A preliminary July 2025 report from MIT's NANDA initiative found most enterprise generative-AI pilots had not yet shown measurable business impact. Its evidence is mostly interviews and surveys (52 organisations interviewed, 153 senior leaders surveyed), and its authors call the figures directional. The throughline across both is that AI's felt value and its measured value can diverge sharply, and only one of them shows up in delivery dates or the income statement.

What this means for your team

  • Don't trust the vibe. If you justify AI tooling on productivity, measure it with something like an evaluation harness; the people using it are not reliable narrators of their own speed.
  • Run a cheap version of METR's design. Split similar tasks AI / no-AI across a sprint or two and compare real completion times, not survey sentiment. Our explainer on the METR study walks through the design in more detail, and what AI is doing to entry-level jobs looks at the same question from the hiring side.
  • Target where AI plausibly helps. Onboarding to unfamiliar code, scaffolding, test generation, not deep edits to systems your seniors already hold in their heads.
  • Separate satisfaction from throughput. Developers can genuinely enjoy the tool while it slows them down; both can be true, and only one shows up in delivery.

The broader lesson generalises well beyond code: feelings about AI productivity are not evidence of it. Building a habit of measuring AI ROI against a real baseline is the only way to know which side of this study your team is on. If you want help setting that measurement up, our team does this kind of work.

A note on what this is not

It would be easy to weaponise this study into "AI tools are a waste of money," and that would be just as unevidenced as the hype it corrects. The honest reading is narrower and more useful: the productivity benefit is real in some contexts, absent or negative in others, and, critically, invisible to the person experiencing it. That means the worst possible way to decide whether to adopt AI tooling is to ask your developers how it feels. They will tell you it's faster, sincerely, and they may be wrong. The right way is measuring your own team's AI-assisted output against a baseline, accepting that the answer might differ by task type and seniority, and letting the data decide where the tool earns its place. Treating a single RCT as gospel is a mistake; treating developer enthusiasm as a metric is a bigger one.

Sources

Written by Zain Ali

Frequently asked questions.

It depends on the setting. METR's July 2025 randomised trial, the best-known controlled test on experienced developers in their own mature codebases, found that allowing AI increased task completion time by 19%, while the developers believed it had cut their time by 20%. Earlier controlled studies on simpler or enterprise tasks found speed-ups, and in February 2026 METR said developers are likely more sped up by today's tools than in early 2025.

METR followed 16 experienced developers through 246 real tasks on mature repositories they had contributed to for about five years on average, in codebases of over a million lines. Tasks were randomly assigned AI or no-AI, the same design used in drug trials. Because these were real issues in code the developers knew intimately, the design removes the usual confound that people reach for AI on easy tasks and grind through hard ones by hand.

Before the results were in, economists and ML experts predicted tasks would take 38 to 39% less time. METR's 2025 trial measured a 19% increase in completion time instead. Everyone was wrong in the same direction, which is part of why the study is worth taking seriously.

METR lists several contributing factors rather than one cause, including the low reliability of the AI output (developers accepted fewer than 44% of generations). One plausible reading: on code you know deeply the bottleneck isn't typing, it's understanding. The AI produces plausible code quickly, but you then have to read it, check it against context the model didn't have, and fix the parts that are subtly wrong, and that review-and-repair loop can cost more than writing the change yourself. It feels fast because the screen fills with code almost instantly: the effort moves from producing to verifying, and verifying is quieter work that doesn't register as labour the same way.

Target it where the human lacks context: onboarding to unfamiliar code, scaffolding, and test generation, rather than deep edits to systems your seniors already hold in their heads. For unfamiliar languages, boilerplate and exploratory prototyping, or for developers new to a codebase, the picture may look very different from METR's result.

Keep reading

All articles
Engineering

Claude Code runs with more autonomy than Claude's chat app, even on the identical model. Anthropic's own data says that's the interface talking.

7 min read
AI Strategy

57% of teams have an AI agent in production. LangChain's own survey found 29.5% of them never check whether it's any good.

7 min read
Security

Does the DOJ data security rule apply to your offshore developer subcontractor? Mostly it turns on where one person sleeps

9 min read