A Pose Encoder Trained on One Sign Language Retrieves Signs Across Six Others It Has Never Seen

Metric-learned ASL encoder, frozen, hits 36-81% top-1 retrieval on six unseen sign languages with one enrolled clip — beating DTW and graph-network baselines cold.

Imagine you learned to identify dog breeds by studying only golden retrievers, poodles, and German shepherds in one country — then someone hands you photos of breeds you've never seen from six different countries and asks you to match each one to a single reference photo in a book. That's the structural problem this paper solves for sign language: train a pose encoder on American Sign Language, freeze it, and ask it to retrieve signs from Turkish, Russian, Indian, Argentinian, Pakistani, and Sri Lankan sign languages it has never encountered. The committed claim: a metric-learned pose encoder trained solely on ASL-Citizen generalizes frozen to six typologically diverse sign languages under extreme low-resource conditions (one or four gallery clips per sign), outperforming both learning-free baselines and a graph-network encoder also trained on ASL. The best configuration hits 68.0% one-shot mean accuracy across six targets, versus 59.0% for the reference encoder and substantially less for angle-based dynamic time warping (16.0–69.6%) and OpenHands graph networks (39.3% six-target mean). The architecture is a pose-based metric-learning encoder using per-pair InfoNCE as its training objective — a contrastive setup where each positive pair gets its own denominator rather than sharing one across the batch. The key structural finding is that blocking attention between body, left hand, and right hand articulators inside the encoder raises accuracy by up to 5.1 points. This is counterintuitive: you'd expect letting body parts attend to each other would help. Instead, forcing the encoder to learn independent articulator representations creates more transferable features. Embedding geometry (spherical, Euclidean, hyperbolic) barely matters — at most 1.6 points across eight seeds — except that hyperbolic token-attention with InfoNCE is consistently 1.4–2.0 points worse. The evaluation is honest in one important way and incomplete in another. The signer-exclusion protocol — where query signers are removed from the gallery when identities are available — prevents the encoder from shortcutting via signer identity rather than sign content. That's a real methodological care. But the six target datasets vary wildly in difficulty and vocabulary size, and the CISLR result (17.3–19.0% on 4,765 Indian signs) shows the method struggles at dictionary scale, barely clearing the RGB baseline's 16.8%. The crossed ablation over embedding geometry, aggregation, training objective, and articulator attention is thorough — but all evaluation is same-team, no independent replication exists, and benchmarks were not pre-registered. The milestone question is sharp: at what vocabulary size does one-shot pose retrieval become practically useful for a sign language dictionary app? The CISLR result suggests ~5,000 signs is already too many for one-shot retrieval at ~18% top-1. The gap between 68% accuracy on smaller vocabularies (dozens to hundreds of signs) and 18% on thousands defines the scaling wall. Getting to 50%+ top-1 on a 5,000-sign dictionary with one enrolled clip per sign would be the threshold where this becomes a usable lookup tool rather than a proof of concept. The obvious experiment not run: fine-tuning or adapting the encoder on even a tiny amount of target-language data. The entire study is about zero-shot transfer, which is the right scientific question to isolate — but the practical question is how much target-language data moves the needle. Even 50 labeled signs from the target language in a few-shot adaptation protocol would reveal whether the frozen encoder's representations are close enough that minimal supervision closes the gap, or whether the cross-language transfer ceiling is fundamental. My read: this is being saved for the next paper. The zero-shot framing is cleaner and more publishable as a standalone result. This is a solid empirical contribution to an underserved problem — most sign language NLP work assumes annotated corpora that simply don't exist for the vast majority of the world's 300+ sign languages. The ablation is thorough, the baselines are fair, and the signer-exclusion protocol is methodologically careful. But the CISLR result is the reality check: at dictionary scale, we're still far from practical retrieval.