Imagine you're a student with a cheat sheet for an exam. You have fifty index cards, each with a worked-out solution to a different practice problem. When the test starts, you glance at the question, pick the card that looks most similar, and copy its answer. You pass — but here's the thing: one of your cards happens to solve 40% of the test all by itself. Did you actually pick well, or did you just get lucky that one card was broadly useful? Most evaluations of your performance can't tell the difference. That's the core mechanism this paper attacks. The committed claim: current evaluations of visual experience retrieval in robotics conflate library quality with selection quality, and once you disentangle them via exhaustive execution, the retrieval rules look far worse than their headline numbers suggest. One fixed experience chosen with hindsight captures 30–58% of the gap between random selection and an oracle. The per-scene visual selection rules are competing over the remaining 0.07–0.15 in success rate — a sliver that's hard to distinguish from noise. The audit methodology is brute-force but clarifying. The authors execute every stored experience in every query scene across two manipulation tasks, three reuse mechanisms, and libraries of K=3, 10, and 50. This full cross-product means every alternative's outcome is known, so you can decompose any retrieval rule's score into what it picked and what that pick was worth. It's the robotics equivalent of showing your work. The retrieval rules tested are nearest-neighbor selection over five visual embeddings spanning raw pixels to CLIP. The findings are pointed. At K≥10, visual rules concentrate on a single experience 1.5–3× more than an oracle would. Their scores then track that experience's intrinsic quality, not scene-specific matching. Worse, wherever a rule's selections differ significantly from a shuffle that preserves its selection frequencies but randomizes scene pairings, the rule performs worse — not better — for every learned image policy tested. Visual distance can predict whether a specific experience-scene pair will succeed (AUROC up to 0.96), but within a single scene it ranks candidates no better than chance for four of five embeddings at K=50 (AUROC 0.45–0.52). The discrimination that looks powerful in aggregate evaporates when you need it most: choosing among candidates for one specific scene. The architecture here is not a new model — it's a diagnostic protocol. The family is nearest-neighbor retrieval in learned embedding spaces (pixels, ResNet, CLIP), evaluated against oracle and random baselines. The compute property the audit leans on is exhaustive cross-execution: every K×N (experience × scene) pair is physically rolled out, which is expensive but produces ground-truth counterfactuals. The authors acknowledge this is infeasible at scale and propose two cheap proxy reports: the distribution of selected experiences and the success rate of the best single experience in hindsight. The integrity regime is strong for a workshop paper. The authors grade themselves with the toughest possible exam — full counterfactual execution, not just end-task success. Baselines include random selection, oracle selection, and the single-best-experience-in-hindsight, which is exactly the right set. The AUROC analysis honestly reports both the strong aggregate discrimination and the weak within-scene ranking, rather than cherry-picking the flattering number. The main limitation: two manipulation tasks is a narrow testbed, and the reuse mechanisms are specific to imitation-learning-style policies. The field implication is methodological, not algorithmic. If the robotics community takes this seriously, every paper claiming improved experience retrieval will need to report two numbers: how concentrated its selections are and how its best single experience performs. These are cheap to compute and hard to game. The milestone here isn't a performance number — it's adoption of the diagnostic. The paper explicitly argues that the current evaluation norm (end-task success alone) is actively misleading the field about what to build next.
