MERGED PER WEEK
+2.4%
PER WEEK
-4.9%
WEEK
+10.2%
CONFIDENCE INTERVAL
+95%
Imagine you upgrade every chef in a restaurant from good German knives to the best Japanese steel — same kitchen, same menu, same cooks. You'd expect faster prep, cleaner cuts, more plates out the door. But when you count the plates at the end of the night, the number is the same. That's what this paper finds about coding models: the knife got sharper, but the throughput didn't budge. The committed claim is stark: switching from a model that scores 11 points higher on SWE-Bench Pro to one that scores 11 points lower produced no statistically significant change in developer productivity, measured by GitHub pull request merge rates. This is not a survey or a vibe check. It's a natural experiment created when a U.S. export control directive suspended access to Claude Fable 5 for 19 days on June 12, 2026, forcing users onto Opus 4.8. The people stayed the same, the projects stayed the same, only the model changed. The study design is unusually clean for this kind of work. Hua tracked 297 Fable 5 users before, during, and after the suspension against a control group of 8,908 developers who stayed on Opus 4.8 the entire time. A difference-in-differences model with developer and date fixed effects found that Fable 5 access increased 7-day merge rate by 1.4 percentage points — with a 95% confidence interval spanning -1.4 to +4.3 — which is a statistical null. The number of pull requests merged per week showed a similarly negligible +2.4% change with an interval of -4.9% to +10.2%. The paper then extends the analysis across five separate model upgrades in 2026, covering 1,208 developers and 15,442 pull requests. The weighted mean effect was -0.9 percentage points, and all five individual estimates were negative, though none reached significance. This paper's real contribution is methodological. The field has been drowning in benchmark scores as proxies for usefulness, with no one testing the link between benchmark deltas and actual work output. The natural experiment design — an exogenous shock forcing a model switch with no confounding changes in personnel or project scope — is about as close as you get to a randomized controlled trial in observational software engineering research. The diff-in-diff framework with fixed effects is the right tool for this question. The integrity picture is strong on some fronts and has visible gaps on others. The outcome variable — merged pull requests on GitHub — is public, observable, and hard to game. The control group is large (8,908 vs. 297 treated). But the study is not pre-registered, the benchmark gap being tested (11 SWE-Bench Pro points) is specific to one model pair, and the 19-day window is short enough that adaptation effects could mask real differences. If developers didn't fully integrate the replacement model's workflow, the null result could reflect switching costs rather than model equivalence. The paper does not appear to test for this. The bigger question this paper pokes at is whether coding benchmarks measure anything that matters for professional software development. SWE-Bench Pro tests a model's ability to resolve GitHub issues autonomously — a narrow, well-defined task. Real developer workflows involve architecture decisions, code review, debugging, documentation, and coordination with other humans, none of which are captured by the benchmark. The null result here is consistent with the hypothesis that the bottleneck in professional software development is not the model's raw code-generation ability but everything around it. What this doesn't settle is whether larger benchmark gaps — 30, 50, 100 points — would eventually move the needle, or whether the effect simply requires longer than 19 days to materialize. The paper is a sharp first cut at a question the entire AI-coding industry has been conspicuously avoiding. The answer so far: your expensive new model might not be buying you anything your old one wasn't already delivering.
The analogy
Upgrading every chef's knife from good German steel to the best Japanese steel — same kitchen, same menu, same cooks — and counting the plates at closing. The knife got sharper, but the number of plates didn't change. The bottleneck was never the blade.
Kitchen knives
Sharper knives should produce more plates per shift, but the bottleneck is coordination, menu complexity, and cook skill — not cutting speed
Coding models
Higher-scoring models should produce more merged PRs, but the bottleneck is architecture, review, coordination, and debugging — not code generation
Honest assessment
This is the first real causal test of whether coding benchmark improvements translate to developer productivity, and the answer is a well-powered null. The design is clever and the diff-in-diff execution is clean. The main vulnerability is the 19-day window — switching costs and adaptation could mask real effects. But the five-upgrade replication, all showing negative point estimates, makes this harder to dismiss. Anyone marketing coding models on benchmark points now has a concrete empirical challenge to address.
What Is CLAIMS?
70
/ 100
BedrockNews applies a structured analytical framework called CLAIMS to academic papers and technical research releases.
How CLAIMS worksClaim
4/5
Ladder
3/5
Architecture
3/5
Integrity
3/5
Milestone
4/5
Successor
4/5
What is this paper actually committing to?
Genuine first: no prior study used an exogenous shock to test whether coding benchmark gains translate to real developer output. The null result is the claim, and it's well-committed.
The field fight
Do improvements on coding benchmarks translate to measurable gains in real developer productivity?
Side A
Higher benchmark scores reflect genuine capability improvements that help developers ship more and better code
Held by: Anthropic, OpenAI, model marketing, SWE-Bench leaderboard culture
Side B
Benchmarks measure narrow autonomous task completion; real dev productivity bottlenecks are elsewhere (architecture, review, coordination)
Held by: This paper, software engineering researchers, skeptics of AI-productivity hype
Milestone tracker
Unlocks: If null persists at 30+ points, coding benchmarks lose credibility as productivity proxies; if signal appears, establishes a minimum effective dose
Current
11
SWE-Bench Pro points
Fable 5 vs Opus 4.8 gap
Next
30+
SWE-Bench Pro points
larger gap test
Requires either a bigger exogenous shock or a prospective RCT — likely 1-2 years
Verified Facts (8)
On June 12, 2026, a U.S. export control directive banned access to Claude Fable 5 for 19 days
Opus 4.8 is 11 points lower on SWE-Bench Pro compared to Fable 5
297 Fable 5 users tracked before, during, and after the suspension
8,908 developers who stayed on Opus 4.8 all the time served as the control group
Five model upgrades covered 1,208 developers and 15,442 pull requests
Fable 5 increased the 7-day merge rate by only 1.4 percentage points [-1.4; +4.3], not statistically significant
Weighted mean across upgrades: -0.9 percentage points [-1.9; +0.2]; all five estimates were negative
Number of pull requests merged per week: +2.4% [-4.9%; +10.2%], not significant
Practical Implications (4)
Engineering leaders paying premium for the highest-scoring coding model may not be getting measurable productivity returns over competent alternatives
Benchmark scores (SWE-Bench Pro) should not be used as sole procurement criteria for coding AI tools without validating against actual team output
The bottleneck in professional software development likely lies in coordination, architecture, and review — not raw code generation capability
Teams switching models due to outages or policy changes can expect minimal productivity disruption, at least over 2-3 week windows
More latest news
Read all