OpenAI spent two years pushing frontier models up the SWE-bench Verified leaderboard, the benchmark it helped build specifically to filter out unreliable test cases. In February 2026, its own frontier evals team published an audit of the benchmark's hardest problems and found that most of what looked like model failure was actually a broken test. A separate study, accepted at ICSE 2026, ran an independent check on the same benchmark and found the opposite failure mode: patches the benchmark counted as solved that weren't. If your team is choosing or trusting an AI coding agent based on a benchmark score, both papers say that score measures something other than what you think it measures.
What OpenAI's own audit found
OpenAI introduced SWE-bench Verified in 2024 as a cleaned-up subset of the original academic SWE-bench, curated with the help of 93 professional Python developers who triple-annotated 1,699 candidate tasks to whittle it down to 500, specifically to remove tasks that were unsolvable due to bad tests rather than a model's shortcomings. In its February 2026 post, "Why SWE-bench Verified no longer measures frontier coding capabilities," OpenAI's evals team explained why they were retiring the number anyway. The audit looked at 138 of those 500 tasks that OpenAI's own o3 model failed to solve consistently across 64 independent attempts on each one. Every task got an independent review from at least six experienced software engineers, with anything a reviewer flagged getting a second pass from another team. The result: 59.4% of the 138 tasks had material issues in the test itself, not in the model's patch. Of that 59.4%, 35.5% were "narrow" tests that enforced one specific implementation and rejected a functionally correct alternative, 18.8% were "wide" tests that checked for functionality the problem description never asked for, and the remaining 5.1% had other defects.
That 59.4% figure describes the hardest tail of the benchmark, the tasks OpenAI's own frontier model already struggled with, not a random sample of all 500. It shouldn't be read as "59% of SWE-bench Verified is broken." What it does show is that the specific tasks used to argue a model is closing in on human-level engineering ability carry more test-design noise than the published leaderboard number lets on. OpenAI's second finding compounds the first: on tasks models did solve, frontier models were reproducing verbatim gold patches and problem-specific details that exist only in the original training data, meaning part of a "solved" score is memorization of that particular benchmark, not general coding ability.
An independent study found the same pattern, from the other side
OpenAI's audit is the benchmark's own steward finding it's too strict in places. A separate group of academics found it's also too lenient. The paper, by You Wang, Michael Pradel and Zhongxin Liu and accepted at ICSE 2026, examined patches produced by three open-source coding agents (CodeStory, LearnByInteract and OpenHands) against SWE-bench Verified's same 500 tasks. Running each patch against the full developer-written test suite, rather than just the smaller slice SWE-bench actually executes, found that 7.8% of patches the benchmark had scored as correct actually failed the real tests, cutting the reported resolution rate by 4.5 percentage points on average across the three agents. The researchers then built PatchDiff, a tool that uses an LLM to compare a submitted patch's behavior against the original developer's reference patch on inputs neither test suite checks, and found 29.6% of "plausible" patches behaved differently from the reference solution. A manual check of a 30% sample of the flagged cases (77 patches) confirmed 28.6% were genuinely incorrect, which extrapolates to roughly 11% of everything the benchmark marked "solved" actually being wrong.
Two groups, two different methods, aimed at opposite failure directions (too strict, too lenient), land on the same conclusion: passing SWE-bench Verified's tests and being a correct patch are not the same event, often enough that neither the leaderboard score nor a "solved" label can stand in for the other.
Why a leaderboard score and real capability keep coming apart
A fixed, finite test suite can't cover every way a patch could be right or wrong, so it fails in both directions: rejecting valid answers it never anticipated and accepting invalid ones it never checked for. Training exposure compounds that. Once a benchmark is public and popular enough to matter, its exact tasks and reference solutions end up in scraped training data somewhere, so a newer model's improved score partly reflects having seen more of the answer key, not a jump in ability. Neither of these is a flaw a better version of the benchmark permanently fixes. The same forces (finite test coverage, training-data leakage, and an optimization target that gets gamed once it becomes valuable enough to chase) will apply to whatever replaces SWE-bench Verified next.
What this costs a team that trusts the number
Translate that into an actual vendor decision. A VP of Engineering comparing two AI coding agents sees one claiming 94% on SWE-bench Verified against another's 89% and treats the five-point gap as a real capability difference worth paying for, or worth loosening code review over. Per both papers above, some real share of that gap could be test-suite quirks or differences in training-data exposure between the two vendors, not a difference in how either agent handles your own, private codebase, which neither number has ever touched. The cost lands later: a team that approved lighter review on agent-generated pull requests because "it's a 94% model" finds out the agent's actual error rate on its own code only after something breaks in production, at which point the fix costs a lot more than the review step would have.
A worked example
Picture a 40-person Series B product company piloting three coding-agent vendors before signing an annual contract. The evaluation deck ranks the options by public SWE-bench Verified score, with one point separating the top two. Engineering leadership picks the top-ranked vendor and, to justify the contract's cost internally, drops the required review pass on any pull request the agent opens against non-critical services. Three weeks in, a "solved" refactor from the agent passes CI clean the same way a narrow or wide SWE-bench test would pass: it satisfies the visible checks without touching every place the original logic was actually called from. It ships. The bug that surfaces two weeks later is a duplicate implementation nobody who reviewed the PR had reason to go looking for, exactly the kind of gap a fixed test suite is structurally unable to catch, because it was never asked to check that specific interaction. The fix isn't picking the other vendor. Neither vendor's number was ever evidence about this codebase.
What actually replaces a leaderboard number
Build a small evaluation set from real, closed tickets in your own repository (10 to 30 tasks is enough to start) and score any agent or model change against that before rollout, the same discipline our guide to building an eval harness for AI features argues has to exist before you trust a model's output at scale. Treat a vendor's benchmark score as a screening filter, not a purchase decision, the way you wouldn't hire a senior engineer off one coding-test score without a work sample against your own codebase. And watch for the same perception gap our piece on measuring whether AI actually makes developers faster found in METR's randomized trial: what looks like speed, or a good score, and what a clock or a full test suite actually shows can point in opposite directions. Here's the honest exception: if an agent is writing low-stakes internal tooling, running in a sandbox, or shipping a small feature where a bad pull request costs an hour to catch, a leaderboard score is a perfectly fine first filter, and a custom eval harness is overkill. Build one to the degree that a bad output can actually reach a customer or your production database, which is a different bar than the one most vendor pitches are selling against.
If you're standardizing on a coding agent, or already have one merging code your team isn't fully reviewing, that's worth a second look before a benchmark score turns out to have been the wrong evidence for the decision. Our Silicon Valley team builds these evaluation sets as part of bringing a staff-augmented engineer into an existing codebase, alongside the maintainability review our piece on AI code quality data covers from the other angle. Tell us what you're evaluating and we'll scope the eval work honestly, as its own task, not folded into an open-ended retainer.
Sources
- OpenAI: Why SWE-bench Verified no longer measures frontier coding capabilities (published February 2026)
- OpenAI: Introducing SWE-bench Verified (2024; 500 tasks verified by 93 Python developers)
- You Wang, Michael Pradel, Zhongxin Liu: Are "Solved Issues" in SWE-bench Really Solved Correctly? An Empirical Study (accepted ICSE 2026)