Russian Sentence Embeddings Without the Transformer Tax: Sign Bits, Mean Pooling, and 11 MiB

StaRSE strips Transformer inference out of sentence encoding by baking token representations into a sign-coded lookup table. 18× faster, 5× smaller, within striking distance of contextual models.

Imagine you run a busy restaurant kitchen and every dish requires a head chef to personally assemble it from scratch — that's Transformer inference for sentence embeddings. Now imagine the chef pre-slices and pre-portions every ingredient into labeled bins at the start of the day, so line cooks can just grab and plate. You lose some of the chef's improvisational flair, but orders fly out 18× faster and the walk-in fridge shrinks to a fifth its size. That's StaRSE. The committed claim: you can distill a ruBERT-base Transformer into a static token table where each embedding dimension is reduced to a single sign bit (plus a separate scalar norm per token), then encode sentences by mean-pooling over norm-scaled sign directions — no Transformer forward pass at query time — and still close most of the quality gap to contextual encoders on the RuMTEB benchmark for Russian. The resulting model scores 51.16 on RuMTEB task mean, 7.98 points above the strongest included static baseline, packaged in 11.3 MiB with 0.08 ms mean CPU latency per sentence versus 55.7 MiB and 1.43 ms for the nearest contextual competitor. The architecture sits squarely in the static-embedding family — think word2vec or GloVe, but modernized with three tricks. First, initialization from a pretrained Transformer (ruBERT-base) rather than from scratch. Second, a multi-stage fine-tuning pipeline using adjacent-sentence pairs and task-shaped training data for Russian. Third, and most distinctive, a contrastive post-training stage that directly optimizes the sign-projected token table — the training loss sees the same binarized representations the deployment system will actually use, closing the train-deploy mismatch that plagues naive quantization. Token norms are stored separately, so the sign bits capture direction and the norms capture magnitude, reconstructing a rough approximation of the original dense vectors. On the ladder, this is an honest step forward for static Russian embeddings, but context matters. The 51.16 RuMTEB score beats static baselines convincingly (+7.98 points), yet the abstract implicitly concedes the contextual encoder it compares against still wins on quality — they frame the comparison in terms of size and speed, not accuracy parity. The closest contextual encoder is not named with its RuMTEB score in the abstract, which is a soft flag: the gap likely remains meaningful. Classical static embeddings (fastText, etc.) are the real baselines being demolished here, not SOTA contextual models. Integrity is reasonable for a workshop paper. RuMTEB is a community benchmark, not a bespoke evaluation. Ablations cover adjacent-sentence supervision, token norms, and sign-projected post-training — the three load-bearing design choices — which is the right set of things to ablate. The paper is accepted to ReNeuIR at SIGIR 2026, a reputable venue. What's missing: no code release is mentioned in the abstract, no independent replication, and the comparison to the contextual encoder uses estimated rather than measured footprint numbers ('estimated 55.7 MiB 16-bit parameter footprint'), introducing some squishiness. The milestone question is clear: can sign-coded static encoders close the quality gap to contextual models entirely, or is there a hard ceiling? The current delta is unstated but implied to be several points on RuMTEB. The next concrete target would be matching a contextual encoder's RuMTEB score while staying under, say, 15 MiB and sub-0.1 ms latency. If the sign-bit ceiling proves real, the interesting successor is hybrid architectures — a static encoder for most queries with a contextual fallback for hard cases. The authors likely didn't run this because it's a systems paper, not a modeling paper, and the hybrid routing logic is a separate research problem. For practitioners: if you're running a Russian-language retrieval or semantic search pipeline on CPUs — think edge deployment, high-refresh-rate indexing, or budget-constrained serving — this is directly useful. The 18× latency reduction and 5× size reduction are real operational wins. If you're on GPUs with a quality-first mandate, this isn't for you yet.