Robot Memory Systems Count Clock Ticks, Not Events — And It Breaks Them Badly

Memory-augmented robot policies use elapsed time as a proxy for task progress. A simple pause during counting tasks crashes success from 94% to 20%. The fix isn't retraining — it's indexing memory by events.

Imagine you're reading a book and someone tells you you're halfway through because half the time has elapsed. But you spent twenty minutes staring out the window. You're not halfway — you just sat there. That's what's happening inside memory-augmented robot policies, and this paper catches them in the act. The claim is clean: memory-augmented vision-language-action (VLA) policies that sample frames uniformly over time and stamp them with time steps learn a "temporal shortcut" — elapsed time predicts progress in training demonstrations because demos proceed at a steady pace. The robot never learns to distinguish waiting from working. The authors call the resulting failure mode "phantom progress," and they demonstrate it with surgical precision. The experimental design is elegant in its simplicity. On RoboMME counting tasks, they introduce a brief pause or wrist fidget — a null event that changes nothing task-relevant. With memory active, success plummets from 47/50 to 16/50 on one task and 44/50 to 10/50 on the other. When the same frames are withheld from memory, success returns. That's a clean ablation: the interruption itself isn't the problem. Memory is. The diagnostic goes deeper than the failure mode. They show that simply rescaling time stamps — making the clock "run fast" — causes the policy to stop early, and slowing the clock causes it to repeat actions. This confirms the policy has internalized elapsed-time-as-progress, not event-completion-as-progress. But here's the surprising finding: removing time encoding entirely doesn't fix things, because the null-event frames still occupy slots in the uniformly-sampled memory buffer. The memory's structure carries temporal information even without explicit timestamps. The vulnerability pattern across RoboMME's released memory designs tracks with how memory is indexed. Time-indexed memories fail; event-indexed memories are more robust. Of the attempted fixes, training with injected pauses — the obvious curriculum approach — doesn't cure the shortcut and actually impairs normal counting performance. Event-based indexing, by contrast, recovers half or more of the lost success without any retraining. This is a strong result: the right structural fix outperforms the brute-force data fix. The implications extend well beyond counting tasks. Any real-world robot deployment will encounter pauses, interruptions, and variable-speed execution. If memory systems confuse duration with progress, they will fail in exactly the environments they're meant to handle. The paper doesn't just identify a bug — it identifies a category of architectural vulnerability that will recur across any time-indexed sequential memory system. What's missing is breadth. The evaluation is confined to RoboMME counting tasks — a narrow slice of robot behavior. The event-based indexing fix recovers "half or more" of the loss, not all of it, and the paper doesn't deeply explore what the remaining failure modes look like. But as a diagnostic paper — one that names a failure, traces its mechanism, and proposes a structural direction — this is sharp, honest work.