You know how deleting a file on your computer doesn't actually erase the data — it just removes the pointer in the directory, and the bits sit there until something overwrites them? LLM agents with persistent memory have the same problem, except worse: the information isn't just lingering in slack space, it's been actively copied, summarized, and recombined into derived artifacts the agent created from the original record. Delete the source, and the ghosts live on in its children. The committed claim: deleting an original memory entry from a fixed-weight LLM agent does not reliably remove the information, because derivative memory artifacts (summaries, recomposed facts) carry forward enough signal to reconstruct the deleted secret. This is demonstrated with a controlled behavioral benchmark — not a theoretical argument — using a never-exposed control group that scores zero recovery, isolating the causal chain from original exposure to post-deletion leakage. The benchmark design is the real contribution. AfterDeleteBench creates synthetic secrets, exposes one agent copy while keeping a matched control unexposed, deletes the record from the exposed agent, then queries both. The exposed-then-deleted Qwen2.5-7B-Instruct agent recovers the secret roughly two-thirds of the time on a single query. The never-exposed control: zero. That counterfactual design is what elevates this from "we noticed a problem" to "we measured a problem with attribution." Critically, recovery doesn't require a surviving verbatim copy of the secret — when only separately stored components remain under a known composition rule, exact reconstruction succeeds in 62–99% of cases across Qwen, Mistral, and Phi-3. The architecture story here is about agent memory systems, not model internals. This is not a weight-space unlearning paper. The models are fixed-weight; the problem lives entirely in the external memory layer and the agent's ability to compose fragments. The paper treats the agent-plus-memory-store as the unit of analysis, which is the right framing for deployed systems. They also surface a subtle measurement issue: the response format and parser alignment can cause you to misclassify a successful recovery as a failure, meaning naive audits undercount leakage. On mitigation, the paper tests two approaches. An exact-literal write filter blocks reinsertion of the secret string but cannot erase information already distributed across derived records — it's a lock on the front door when the information already left through the windows. Provenance-aware closure, which traces and removes all descendants of the deleted record, eliminates observed recovery in the tested settings. That's the right architectural direction, but the paper is honest that this was tested only in positive-baseline configurations, not adversarial ones. The integrity profile is solid for a first benchmark paper. The counterfactual (never-exposed control) is the strongest design choice — it converts correlation into attribution. Three model families tested (Qwen, Mistral, Phi-3) adds breadth. The limitation is that all evaluation is the authors' own benchmark on their own synthetic secrets in their own memory system design. No independent replication, no pre-registration, no adversarial red-teaming of the mitigation. The 62–99% range across models is wide enough to suggest the result is real but model-dependent. The gap in the successor space is obvious: real-world agent memory architectures. MemGPT, AutoGen, LangChain memory modules — the paper tests a controlled synthetic memory system, not the messy graph stores and retrieval pipelines deployed in production agents. The honest read is probably (a) and (c): testing on production memory systems is a different engineering challenge, and it's the natural next paper. The other missing experiment is adversarial probing — what happens when someone actively tries to reconstruct a deleted secret using prompt injection or indirect queries, rather than a single direct ask?
