Imagine you hand someone a restaurant review — not the recipe, not the ingredient list, just the review — and ask them to recreate the dish. If they've eaten at that restaurant a thousand times before (memorized the training data), they'll nail it. If they haven't, the dish tells you something real about their culinary reasoning. That's the core tension in ReproBAIT: when an LLM coding agent rebuilds a biological AI tool from a paper description, are you measuring genuine scientific reasoning or sophisticated recall? The committed claim: coding agents can design, implement, and train biological AI tools from limited public descriptions, even without internet access, posing a real threat to managed-access biosecurity strategies and commercial IP. The authors test this across three tools of escalating complexity — ProteinMPNN (inverse protein folding), RFdiffusion (all-atom generative diffusion), and an unnamed closed-source molecular design tool — giving agents training data, compute budgets, and varying levels of information about what to build. The results split cleanly by complexity. For ProteinMPNN, multiple agents achieve substantial reproduction. For RFdiffusion, reproduction is partial. For the closed-source tool, capability is limited. This gradient is the most informative part of the paper: it tells you that current agents can reconstruct moderately complex ML pipelines but hit a wall at highly complex architectures. The ladder here is honest — the authors aren't claiming agents can rebuild anything, they're mapping the frontier of what's rebuildable. The memorization problem is the elephant in the room, and the authors deserve credit for tackling it head-on rather than ignoring it. ProteinMPNN and RFdiffusion are open-source with public codebases — almost certainly in every frontier model's pretraining corpus. The authors institute a memorization-checking methodology and find "moderate evidence consistent with memorisation, varying across coding agents." This is the right finding to report, but it substantially clouds the headline result. How much of the reproduction is reasoning vs. retrieval? The paper can't fully answer this, which is why the closed-source tool result — where memorization is ruled out — becomes the most scientifically valuable data point, despite showing limited capability. The evaluation architecture is sandboxed (no internet), time-bounded, and compute-bounded, with held-out test data for scoring. This is a thoughtful setup. The information access conditions are varied — from just a paper abstract to more detailed descriptions — creating a gradient that maps how much description leakage is enough to enable reconstruction. The security framing is legitimate: if a brief public description plus pretraining exposure is sufficient, then managed-access regimes that publish papers but withhold code may be less protective than assumed. The dual-use implications are the paper's real contribution to policy. The technical result — agents can rebuild some tools — is interesting but expected. The policy inference — that publishing descriptions of sensitive tools may be operationally equivalent to publishing the tools themselves — is the finding that should change behavior. Tool developers releasing papers about dual-use capabilities now face a concrete threat model: an adversary with a frontier coding agent and your paper abstract may be closer to having your tool than you thought. What's missing is scale. Three tools, with the most informative one (closed-source) showing the least capability. The memorization confound remains unresolved for the open-source tools. And the paper doesn't benchmark against human teams — how long would a competent ML engineer take to reproduce these tools from the same descriptions? That human baseline would transform the threat assessment from theoretical to operational.
