Imagine you're assembling a jigsaw puzzle, but someone keeps dumping duplicate copies of the same pieces onto your table every few seconds. You'd drown in redundancy — not because the puzzle is hard, but because you can't tell which pieces you've already placed. The obvious fix: before adding new pieces, check whether you already have one covering that spot. That's the core mechanism of TrackEverything: instead of tracking every pixel independently across hundreds of frames (which blows up memory), it lifts the video into 3D world coordinates and deduplicates observations of the same physical surface using voxelization at sliding-window boundaries. The committed claim: TrackEverything is the first 3D point tracker that can densely track all visible points across videos exceeding 1,000 frames within 40 GB of GPU memory. Prior systems had to choose — track everything but only on short clips, or track sparse queries over long horizons. This paper says you can have both, by decoupling computational cost from video duration and coupling it instead to the number of unique 3D surfaces in the scene. Three architectural moves make this work. First, voxel-based deduplication merges co-located tracks at sliding-window boundaries, so observing the same wall from 500 frames doesn't cost 500× more than observing it once. Second, a two-stage pipeline: an endpoint refiner predicts where each point ends up and classifies it as static or dynamic, then a lightweight trajectory refiner fills in dense motion paths only for dynamic points — static geometry is cheap. Third, 3D WAFT replaces the memory-hungry 4D correlation volumes used in prior work with efficient feature sampling directly in the scene point cloud. On the TAPVid-3D benchmark, TrackEverything outperforms all open-source all-frame dense 3D trackers by more than 20 percentage points in Average Position Distance (APD) on short clips. On long sequences, it remains competitive with state-of-the-art sparse trackers despite tracking far more points. The gap between dense and sparse tracking on long videos has been a persistent pain point; this paper substantially narrows it. The ladder position is strong but honest. The 20%+ APD margin is measured against open-source dense 3D trackers — meaning the comparison class is other all-frame, all-point methods, not sparse trackers that pick their battles. Against sparse SOTA on long sequences, the paper claims competitive (not superior) performance. This is the right framing: sparse methods are playing a different game, and matching them while tracking everything is the actual achievement. Integrity is reasonable for a computer vision paper. TAPVid-3D is an established community benchmark, and the comparisons are against current open-source methods. However, there's no pre-registration, no independent replication, and the 'competitive with sparse SOTA' claim on long sequences deserves scrutiny — 'competitive' is doing real work in that sentence without hard numbers in the abstract. Code and model availability isn't stated in the abstract, which is a yellow flag for reproducibility. The obvious next experiment they didn't run: real-time or near-real-time inference on egocentric video (think AR headsets or robotic manipulation), and scaling to truly open-world scenes with many independently moving objects. The 40GB memory ceiling and 1,000-frame horizon are impressive but still lab-scale for continuous perception. The honest read is (a) — they likely ran out of compute for the scaling experiments that would prove real-world viability, and those experiments are the natural sequel paper.
