AI-Text Detectors Work Great in the Lab — Then Students Show Up and Start Copy-Pasting Half a Sentence

First study with keystroke-level provenance on real student essays shows detectors collapse on partial AI use — the exact pattern that dominates authentic classrooms.

Imagine you're a customs inspector trained to spot counterfeit bills. You're excellent at it — 96% accurate — because every fake you've ever seen was printed on a completely different press than the real thing. Now someone hands you a bill where the serial number is genuine but the watermark is photocopied. Your training never covered that. That's the core problem this paper surfaces for AI-text detection. The committed claim: current AI-text detectors, including top-ranked shared-task systems, perform well on the binary problem (fully human vs. fully AI text) but degrade sharply on the mixed-provenance writing that actually shows up in classrooms. Using PISA 2025 Foreign Language Assessment data from 15-year-olds in Germany writing English argumentative essays, the authors had something almost no prior study has: complete keystroke, clipboard, and chat logs that let them trace every character in the final essay to its origin — human-typed, AI-generated, or mixed. Two detector architectures were tested. A feature-based detector (adapted from a top-ranked shared-task system) relies heavily on lexical complexity signals — longer words, more sophisticated syntax. It hits macro-F1 of 0.96 at the text level and 0.95 at the sentence level for the clean binary case. But a trivial adversarial move — asking the chatbot to 'rewrite this like a student would' — craters AI recall from 0.92 down to 0.60 with gpt-4o-mini and 0.41 with Qwen3. The signal the detector learned was vocabulary sophistication, and that signal is trivially removable. An embedding-based detector using frozen Llama-3.1-8B representations proves more robust for whole texts (AI recall ≥ 0.94 even after rewriting) but drops to as low as 0.59 at the sentence level. The most policy-relevant finding concerns mixed text — essays where students wove AI-generated fragments into their own writing. These fall between the two classes in detector confidence, landing in a classification no-man's-land. Meanwhile, text written after chatbot interaction but without any traceable AI material is classified as human-written about 95% of the time. This means the detector isn't picking up on 'AI-influenced thinking' — it genuinely needs copied text to flag. The validation setup is unusually strong for this subfield. The PISA 2025 FLA dataset provides ground-truth provenance labels derived from logged keystrokes and clipboard events — not researcher-generated prompts or Turk-worker essays. Both detectors are adapted from top-ranked shared-task systems, not strawmen. The authors test robustness against adversarial rewriting with two different LLMs (gpt-4o-mini and Qwen3), and they examine the spelling-error confound explicitly, finding detector performance is independent of it. The architectural lesson generalizes beyond education. Feature-based detectors learn surface statistics that are artifacts of the generation process, not deep signatures — and those artifacts are trivially game-able. Embedding-based detectors capture something more structural but lose resolution at finer granularities. Neither architecture currently handles the mixed-provenance case that defines real-world AI assistance. The gap between 'can detect fully-AI text' and 'can detect partial AI use in authentic writing' is the gap that matters, and this paper is the first to measure it with real provenance data. The field fight here is between detection-as-binary-classification (the shared-task paradigm, where you get a clean human/AI label) and detection-as-provenance-tracing (where you need to locate which parts of a text are AI-generated). This paper argues, with strong evidence, that the binary paradigm is insufficient for real deployment. The next milestone is obvious: a detector that can reliably flag mixed text at the sub-sentence level with provenance awareness. We're not there yet, and this paper makes clear exactly how far away we are.