
MODEL SCORED APPROXIMATELY
+83%
COURSE MATERIALS
+85%
BENCHMARKS. SHOWED
-90%
GPT-4O STILL HIT
-85%
Imagine you're a locksmith who designs a new lock, then hands it to the three best lock-picks in the world and watches them work. That's what this paper does — except the locks are college exam questions and the picks are GPT-4o, Claude 3.5, and Gemini 1.5 Pro. The result: even the locks designed specifically to resist picking got picked most of the time. The committed claim: across 75 real college assessment questions from 17 disciplines — 61 of which faculty intentionally designed to be LLM-resistant — state-of-the-art LLMs still score approximately 83% without supplementary course materials and 85% with them. The five faculty strategies for AI-proofing (personal reflection, discipline-specific jargon, multi-step reasoning, real-world application, and ambiguity/open-endedness) produced measurable but insufficient resistance. This is not a standardized-test study or a single-discipline slice; it's the broadest cross-disciplinary audit of AI-proofing attempts to date. The ladder here is awkward because the paper isn't competing against a prior method — it's measuring a failure rate. The most useful comparison is against prior LLM-on-exam studies (Kung et al. 2023 on USMLE, OpenAI's own GPT-4 technical report benchmarks). Those showed 70-90% on standardized exams not designed to resist AI. This paper shows that intentional resistance barely closes the gap: the best model (GPT-4o) still hit 83-85%. The delta between 'AI-proofed' and 'not AI-proofed' questions is the real finding, and it's disappointingly small for anyone hoping faculty ingenuity alone can solve this. Architecturally, there's no novel model here — this is an evaluation study. The three LLMs tested (GPT-4o, Claude 3.5 Sonnet, Gemini 1.5 Pro) represent the current commercial frontier of autoregressive transformer-based models. The experimental design uses a 2×2 structure: questions with and without supplementary course materials, scored by faculty graders using their own rubrics. The thematic analysis of AI-proofing strategies is qualitative, coded via Braun and Clarke's reflexive method. Integrity is mixed. On the strong side: real faculty from 11 universities contributed authentic assessments they actually use, and the questions span 17 disciplines — far broader than typical studies. Faculty themselves graded the LLM responses, reducing measurement circularity. On the weak side: 75 questions is a modest sample, the LLM versions tested will be obsolete within months, there's no pre-registration, and the five AI-proofing strategies emerged from post-hoc thematic analysis rather than a priori coding. The absence of a formal inter-rater reliability check on the thematic coding is a gap. The milestone question is uncomfortable: if 83-85% is where we are with intentional resistance today, and model capabilities are improving at the current rate, the question isn't 'when will AI-proofing work' but 'when does the field accept it won't.' A concrete next number would be: at what score threshold (95%? 98%?) do universities formally abandon take-home assessment as an integrity-bearing format? This paper doesn't name that threshold, but the trajectory points there within 2-3 model generations. The obvious successor experiment is longitudinal: re-run the same 75 questions on next-generation models (GPT-5, Claude 4, Gemini 2.0) and measure the decay rate of AI-proofing strategies over time. The authors almost certainly plan this — it's the natural next paper. The harder experiment they didn't run: testing whether proctored in-person exams with AI-detection actually change student behavior, which would require IRB approval and institutional cooperation at a scale this team may not yet have.
The analogy
Imagine you're a locksmith who designs a new generation of locks specifically to resist the three best lock-picks in the world. You hand them the locks, watch them work — and they open 83% of them anyway. The lock designs weren't bad; the picks just got too good. That's what faculty AI-proofing strategies are facing: the tools they're trying to resist improve faster than the resistance strategies can evolve.
Lock design
A locksmith creates novel mechanisms (pins, discs, sidebars) hoping to defeat known picking techniques
AI-proof exams
Faculty design novel question features (reflection, jargon, multi-step) hoping to defeat known LLM capabilities
Honest assessment
This is a useful empirical snapshot that confirms what most educators already suspect: intentional AI-proofing doesn't work well enough. The cross-disciplinary breadth (17 fields, 11 universities) is the real contribution, not the top-line finding. The small sample (75 questions), lack of pre-registration, and rapid model obsolescence limit the shelf life. Worth reading if you're designing assessment policy now; the specific numbers will be outdated by mid-2026.
What Is CLAIMS?
53
/ 100
BedrockNews applies a structured analytical framework called CLAIMS to academic papers and technical research releases.
How CLAIMS worksClaim
3/5
Ladder
2/5
Architecture
2/5
Integrity
3/5
Milestone
2/5
Successor
4/5
What is this paper actually committing to?
Broad but not first. Cross-disciplinary scope (17 fields, 11 universities) is the contribution, not the finding that LLMs beat exams — that's well-established. The AI-proofing strategy taxonomy adds modest novelty.
The field fight
Can assessment design outpace LLM capability, or must higher education structurally abandon unsupervised evaluation?
Side A
Clever question design (Bloom's higher-order, personal reflection, ambiguity) can stay ahead of LLMs
Held by: Assessment-design reformers, most faculty development offices
Side B
AI-proofing is a losing arms race; universities must move to proctored oral/live assessment or accept AI as a tool
Held by: AI-integration advocates, some CS/engineering departments
Milestone tracker
Unlocks: Formal institutional abandonment of unsupervised take-home assessment as an integrity-bearing format
Current
83-85
% LLM accuracy
this paper (AI-proofed questions)
Next
95
% LLM accuracy
policy-forcing threshold
~2-3 model generations (1-3 years at current pace)
Verified Facts (7)
75 questions from authentic college coursework assessments, spanning 17 disciplines
61 questions were intentionally designed by faculty members to be 'LLM-proof'
14 questions were sourced from online courses
37 faculty members from ten U.S. universities and one U.K. university
Best-performing model scored approximately 83% without supplementary course materials and 85% with them
Three state-of-the-art LLMs evaluated
Five strategies identified through thematic analysis
Practical Implications (4)
Faculty investing time in 'AI-proofing' take-home exams should expect at best a marginal reduction in LLM success rates, not a solution
Institutions should accelerate migration to proctored, oral, or live-demonstration assessments for high-stakes evaluation
The five named AI-proofing strategies (personal reflection, jargon, multi-step reasoning, real-world application, ambiguity) can be used as a checklist, but should not be treated as sufficient defense
Assessment integrity policies should plan for LLM accuracy reaching 95%+ on current question formats within 2-3 years
More latest news
Read all