MOAI Cuts Encrypted Transformer Inference Time in Half by Eliminating the Rotations Nobody Questioned

A new HE packing strategy removes 2,448 costly rotations from BERT inference and halves latency vs. prior art. Extends to LLaMA-3-8B. Code is open.

Imagine you're rearranging furniture in a small apartment. Every time you want to move the couch, you first have to rotate the coffee table out of the way, then rotate the bookshelf, then rotate the rug — dozens of intermediate moves just to get one thing where it needs to go. Now imagine someone redesigns the floor plan so the couch slides straight into place with no intermediate rotations at all. That's MOAI's core trick applied to homomorphic encryption: it redesigns how data is packed inside ciphertexts so that the expensive rotation operations — the furniture-shuffling of HE computation — largely disappear. The committed claim: MOAI achieves a 52.8% reduction in evaluation time over THOR (Moon et al., CCS 2025), the current state-of-the-art for non-interactive HE-based transformer inference, and a 55.7% reduction over Powerformer (Park et al., ACL 2025). It does this not by approximating away transformer components, but by rethinking the packing and evaluation flow so the same math happens with far fewer HE rotations. The reported amortized time is 2.36 minutes per input for BERT-base on a single GPU. The architecture sits squarely in the CKKS (approximate arithmetic) homomorphic encryption family, operating in the non-interactive setting — meaning the client sends encrypted input once and gets the result back, with no back-and-forth. The key structural choices are threefold: a unified column-and-diagonal packing strategy that eliminates format conversions between layers; rotation-free algorithms for Softmax and LayerNorm that replace costly ciphertext rotations with cheaper polynomial operations; and an interleaved batching scheme for ciphertext-ciphertext matrix multiplications. The method leans heavily on GPU acceleration for HE operations, which is where the single-GPU benchmark matters. On the ladder, MOAI names its baselines honestly. Against THOR — the strongest non-interactive HE transformer baseline at submission time — it claims 52.8% faster end-to-end. Against Powerformer, which takes the alternative approach of simplifying the transformer's nonlinear functions before encrypting, it claims 55.7%. The rotation count comparison is granular: MOAI uses at least 1.7× fewer HE rotations across all matrix multiplication stages of BERT-base, and eliminates 2,448 rotations specifically in Softmax and LayerNorm. These are apples-to-apples comparisons within the non-interactive HE setting. The paper does not compare against interactive (client-aided) protocols, which are faster but weaker on the privacy model. Integrity is reasonable but not exceptional. The benchmarks are on BERT-base, a community-standard architecture, and the comparisons are against named 2025 works — not stale baselines. Code is publicly released as open source. However, there's no independent replication, no pre-registration, and the accuracy impact of the Softmax/LayerNorm approximations is reported but not stress-tested across diverse downstream tasks. The LLaMA-3-8B extension is described as showing 'extendibility' — this is a demonstration of applicability, not a full performance benchmark at that scale. The milestone question for this subfield is clear: when does encrypted inference become fast enough that a real service deploys it? At 2.36 minutes per BERT input, we're still orders of magnitude from interactive latency. The next meaningful threshold is probably sub-10-second encrypted BERT inference, which would make it viable for batch processing of sensitive documents (medical records, legal discovery). That likely requires another 10-15× speedup, achievable through some combination of hardware advances (next-gen GPU HE libraries), further algorithmic compression, and possibly hybrid approaches. The obvious experiment not run: scaling the full pipeline to LLaMA-3-8B with end-to-end latency numbers comparable to the BERT benchmarks. The paper mentions LLaMA-3-8B extendibility but doesn't report wall-clock times at the same level of detail. The honest read is (a) — they likely ran out of compute or the numbers aren't competitive enough to headline. Running full encrypted inference on an 8B parameter model with CKKS is genuinely expensive, and the contribution is primarily algorithmic, not about brute-forcing scale.