Imagine you're staring at a river from a helicopter. Someone dumps a truck of dye upstream and everyone on the bank shouts that the river has changed color. But from altitude, you can see: the river was already shifting course for years, and the dye mostly followed existing currents. That's what this paper does for LLMs and science — it zooms out. Bekkerman et al. analyze 2.5 million full-text papers across five disciplines (computer science, biomedicine, physics, social sciences, and mathematics) spanning 2015 to 2025. They track lexical shifts (word frequencies), structural changes (paper length, section counts, citation patterns), epistemological markers (hedging language, methodology terms), and social dimensions (coauthor counts, international collaboration). The question is simple: did ChatGPT's arrival in late 2022 create detectable breaks in any of these trends? The answer is mostly no. Word frequency shifts appeared — some LLM-associated terms spiked — but structural, epistemological, and social trends largely continued on trajectories established well before GPT-3.5 launched. The river was already moving. Among the genuine exceptions: coauthor team sizes in computer science accelerated significantly in Asia post-2023, suggesting LLMs may be lowering collaboration barriers in specific regional and disciplinary contexts. A seeming decline in biomedical topic diversity turned out to reflect standardization in article formatting, not a narrowing of intellectual scope. The study's core method is interrupted time series analysis across its dimensions, looking for statistically significant breaks around the LLM inflection point. This is straightforward applied scientometrics — no exotic architecture, no novel algorithm. The value is in the scale and coverage, not the method. Five disciplines, ten years, 2.5 million papers is a serious corpus. The team builds what they call an "empirical foundation for monitoring the health of global science," which is more infrastructure than breakthrough. The integrity picture is mixed. The corpus is large and spans multiple disciplines, which guards against cherry-picking. But the validation is entirely internal — the team is measuring their own proxies against their own trend models. There's no external ground truth for "has science changed," which means the choice of dimensions is load-bearing. If LLMs are changing science in ways that don't show up in lexical frequencies or coauthor counts — say, by shifting which questions get asked, or by subtly homogenizing reasoning patterns — this methodology would miss it entirely. The honest framing is that this is a well-executed null result at the macro level, with a couple of genuine signals in the noise. It doesn't prove LLMs haven't changed science; it proves the most obvious surface-level indicators haven't shifted dramatically in the first two years. That's useful but limited. The paper is better understood as a monitoring baseline than as a definitive verdict. The authors, to their credit, frame it this way. The successor question is obvious: what about content-level analysis? Tracking whether LLM-assisted papers are more formulaic in their reasoning, whether they converge on similar framings, whether the distribution of novel claims is narrowing — none of that is captured by the lexical and structural features measured here. The authors likely know this is the next frontier but lack the methodology (or perhaps the compute for semantic-level analysis at this scale) to pursue it in this paper.
The analogy
You're watching a river from a helicopter. Someone dumps dye upstream and the crowd below shouts the river changed color. From altitude, you see: the river was already shifting course for years, and the dye mostly followed existing currents. This paper is the helicopter view — measuring whether LLMs actually redirected science's current, or just colored water that was already flowing the same direction.
River from altitude
Tracking the actual course of a river over years to see whether a dye dump changed its direction or just followed existing flow
Scientometric trends
Tracking lexical, structural, and social dimensions of 2.5M papers to see whether LLMs bent pre-existing trajectories
Honest assessment
A solid, large-scale null result that establishes a monitoring baseline for LLM impact on science. It proves the most obvious surface-level indicators haven't shifted dramatically in LLMs' first two years — useful but inherently limited. The real question (whether LLMs are homogenizing scientific reasoning at a semantic level) remains entirely unaddressed. Best understood as infrastructure for future work, not a definitive verdict. Valuable for anyone building policy or institutional responses to LLMs in science who needs to separate signal from hype.
What Is CLAIMS?
60
/ 100
BedrockNews applies a structured analytical framework called CLAIMS to academic papers and technical research releases.
How CLAIMS worksClaim
3/5
Ladder
3/5
Architecture
2/5
Integrity
3/5
Milestone
3/5
Successor
4/5
What is this paper actually committing to?
Well-executed null result at macro scale. Genuine contribution as a monitoring baseline, but 'things haven't changed yet on surface metrics' is a limited claim with a short shelf life.
The field fight
Have LLMs already fundamentally changed how science is practiced, or is the transformation still ahead?
Side A
LLMs are already transforming scientific productivity, writing, and discovery — the revolution is underway
Held by: Microsoft Research, Nature editorials, AI-for-science advocates
Side B
Surface indicators show no structural break yet — the hype has outrun measurable evidence of transformation
Held by: This paper, scientometrics community, skeptical bibliometricians
Milestone tracker
Unlocks: Detecting whether LLMs homogenize reasoning patterns, narrow claim novelty, or converge argumentation — the effects that actually matter
Current
2.5M
papers (surface features)
this paper
Next
2.5M+
papers (semantic analysis)
content-level detection
~1-3 years as LLM-based content analysis scales
Verified Facts (5)
Authors analyze 2.5 million full-text papers across five disciplines from 2015 to 2025
Growth in computer science coauthor teams accelerated significantly in Asia
Apparent declines in biomedical topic diversity largely reflected standardization in article formats
Most pre-2023 trends continued with limited detectable breaks
Authors: Ron Bekkerman, Nichole Li, Bahareh Kargar, Mark Somers
Practical Implications (4)
Policymakers and university administrators citing LLM-driven transformation of science should note that empirical evidence at scale doesn't yet support dramatic claims
Researchers studying LLM impact should move beyond lexical and structural features toward semantic-level content analysis to detect the effects that matter most
The Asia-specific acceleration in CS coauthor teams is a real signal worth tracking — LLMs may be lowering collaboration barriers unevenly across regions
The biomedical topic diversity decline being an artifact of formatting standardization is a cautionary tale about surface metrics masquerading as substance
More latest news
Read all