Imagine you're a restaurant inspector, but instead of visiting kitchens, you're reading Yelp reviews written by the chefs themselves. You notice something odd: almost nobody ever goes back to check if the first review was accurate. When someone does revisit, they change the recipe, the diners, and the lighting — then call it a 'replication.' That's the state of usable security and privacy research, and this paper is the first serious audit of the inspection system itself. The committed claim: across 13 major venues (spanning security, privacy, and HCI) over the period 2016–2025, only 24 papers report replicated user studies — and not a single one is a strict exact replication. Approximately two-thirds of those 24 altered multiple aspects of the original work. The field is building policy-relevant conclusions about how humans interact with security systems on a foundation that has almost never been independently re-checked. The method is mixed but straightforward. The team examined Calls for Papers from all 13 venues for replication guidance, ran a systematic literature search yielding the 24 papers, categorized them using the Olszewski et al. (2025) replication taxonomy, and surveyed the original authors about their motivations. This is a meta-study — it doesn't run experiments, it audits the experiment ecosystem. The architecture is closer to systematic review than to empirical science, which means its strength is coverage and its weakness is dependence on what authors chose to label as replication. Four findings land. First, venue CFPs lack clear guidance for authors and reviewers on how to handle replication work — which means both submitters and gatekeepers are flying without instruments. Second, even when replication papers exist, it's hard to determine what was actually changed versus the original study. Third, zero strict exact replications exist in the sample. Fourth, temporal and contextual changes — the world moved on, users changed, threat landscapes shifted — are the most frequently cited motivation for attempting replication at all. The integrity picture is honest but limited. This is a descriptive meta-study, not a hypothesis-testing experiment. The 13 venues are named, the search methodology is systematic, and the taxonomy (Olszewski et al. 2025) is an external framework. The author survey adds a qualitative layer. But the sample of 24 is small enough that any missed paper shifts the percentages meaningfully, and we don't know how many replications were attempted but never published because venues rejected them — survivorship bias is the elephant in the room. The field fight here is real: should usable security venues actively incentivize replication, or does the current novelty-first publication culture adequately self-correct? This paper lands firmly on the incentivize side. The practical recommendations target venues (update CFPs), researchers (report modifications transparently), and reviewers (evaluate replication on its own terms, not as lesser novelty). These are institutional interventions, not technical ones. The successor experiment nobody ran — and the honest read is (a), resource constraints — is a large-scale survey of venue reviewers and program committee members asking how they actually evaluate replication submissions. The authors surveyed replication authors but not the gatekeepers. That asymmetry matters: knowing why researchers replicate is useful, but knowing why reviewers reject replications would be transformative. That study requires access to PC members across 13 venues, which is a coordination problem a seven-person team likely couldn't solve in one paper cycle.
