Your AI Search Agents Agree With Each Other — That Doesn't Mean They're Right

Formal impossibility results for multi-agent screening: agreement ≠ accuracy, acceptance gives precision but not recall, and the only escape hatch is sampling what agents rejected.

Imagine you hire three locksmiths to check every door in a building. They each try a subset and report which ones are unlocked. If all three agree a door is unlocked, you're confident — but their agreement tells you nothing about the locked doors none of them tried. Worse, even if you personally verify every door they flagged, you only learn your locksmiths' hit rate on doors they noticed, not how many unlocked doors they walked past. That's the core mechanism here: in any pipeline where AI agents filter items for a costly human judge, the judge only sees what agents pass forward, and the unseen remainder is an epistemic black hole. Chang and Zhang formalize what you can and cannot learn about AI screening agents — in agentic search, legal e-discovery, or AI-assisted paper reviewing — when the human assessor only judges items agents surface. The committed claim: agreement among any number of agents establishes concordance, never correctness. Acceptance by the assessor yields precision (how many surfaced items were actually relevant) but only a lower bound on recall (what fraction of all relevant items were found). These are not empirical findings — they are impossibility results. No amount of data fixes the problem. The constructive result is where it gets useful. If you know which agents found which relevant items, you get an upper bound on recall under the mild assumption that agents' finds aren't negatively correlated — they don't systematically avoid each other's hits. You also get a cap on the marginal value of adding one more agent, if agents are interchangeable. This is the diminishing-returns theorem for agent ensembles: past a certain point, throwing more agents at the problem buys you almost nothing. The real escape hatch — and the paper's most practical contribution — is a sampling protocol. To actually pin down recall, you must judge items agents did NOT pass. A random sample of rejected items estimates how many relevant items were missed, to any chosen accuracy. Crucially, the required sample size depends only on the desired confidence and precision, not on the number of agents. Whether you have 2 agents or 200, the audit cost is the same. The paper operates in the tradition of capture-recapture statistics and impossibility results in information retrieval (think of TREC pooling debates). The architecture is purely analytical — no ML, no training, no compute. This is a probability-and-set-theory paper wearing an AI costume, which is exactly its strength. The results are assumption-free where they claim impossibility and assumption-light where they give bounds. Integrity is strong for what this is: mathematical results with proofs. There are no benchmarks to game because these are not empirical claims. The validation is the proof itself. The weakness is the gap between the theoretical framework and messy deployment: real agents have correlated errors, real assessors are noisy, and the independence assumptions underlying the sampling protocol need testing on actual e-discovery or paper-review datasets. The paper's absence is its empirical validation. No simulation, no case study, no deployment data. The authors present clean theory and stop. The honest read: this is deliberate scoping, not evasion — the theory stands on its own — but the obvious next move is to run the sampling protocol on a real TREC collection or an actual e-discovery corpus and show the bounds are tight enough to be useful in practice.