VeilTune Keeps Both the Data and the Model Secret During LoRA Fine-Tuning — A First for Full Backpropagation in MPC

First protocol to run complete LoRA fine-tuning inside secure multi-party computation, hiding adapter weights from data owners while matching plaintext accuracy and beating prior art by 5.3 points.

Imagine you're a chef who wants to learn a restaurant's secret sauce recipe by watching them cook your ingredients. Even if the restaurant encrypts the final dish, you can reverse-engineer the recipe from the intermediate steps — the temperature they set, the order they added spices, the color of the reduction at each stage. That's the core vulnerability VeilTune addresses: existing secure fine-tuning protocols encrypt the final model but leak intermediate training signals — hidden representations, logits, gradients — that let a curious data owner reconstruct the service provider's adapter almost perfectly. The committed claim is stark: VeilTune is the first protocol that runs full backpropagation-based LoRA fine-tuning entirely inside secure multi-party computation, so that neither party ever sees any value that depends on both the data and the adapter. Before this, protocols like SecP-Tuning protected training data but still revealed intermediate computations to the data owner. The authors demonstrate that this is not a theoretical concern — from a single mini-batch of 32 samples' hidden representations, an attacker can fit a copycat adapter that agrees with the original on 99.2% of test inputs. Logits alone yield 94% agreement. The adapter theft threat is real and cheap. Architecturally, VeilTune lives in the two-party secret-sharing MPC family (specifically 2PC over arithmetic shares), applied to the LoRA adapter fine-tuning pipeline on frozen pretrained transformers. The key engineering moves are threefold: compute only the layers the loss actually touches (skipping frozen-weight-only forward paths where possible), replace generic MPC-friendly approximations of nonlinear functions (GELU, softmax, layer norm) with training-aware variants calibrated to the actual value distributions that arise during training, and introduce an oblivious mechanism to detect and skip iterations corrupted by fixed-point truncation errors — a subtle but critical move, since MPC arithmetic uses fixed-point representation where overflow silently poisons gradients. On the ladder, VeilTune competes against SecP-Tuning, the current SOTA for secure prompt tuning. Across four few-shot tasks on RoBERTa-large, VeilTune is 5.3 accuracy points better on average and nearly matches plaintext (unencrypted) training accuracy. On SST-2 specifically, it delivers a 3.5× end-to-end speedup and 3.6× less communication than SecP-Tuning, and 26× and 17× improvements respectively over naively running full fine-tuning in MPC. These are head-to-head numbers on the same tasks, not apples-to-oranges comparisons. Integrity is solid but bounded. The evaluation uses four standard NLP benchmarks (SST-2 among them) on RoBERTa-large, a well-understood model. The comparison is against the named SOTA (SecP-Tuning) with explicit numbers. However, validation is entirely same-team simulation — no independent party ran VeilTune, no pre-registration exists, and no code release is mentioned in the abstract. The 13-page paper with 7 tables suggests thorough reporting, but replication is the obvious gap. The few-shot regime (small training sets) is both a practical constraint and a convenient one — it keeps MPC costs manageable but leaves open how the protocol scales to larger datasets. The milestone to track is scaling: today's result is on RoBERTa-large (355M parameters) with LoRA adapters on few-shot tasks. The next meaningful threshold is running VeilTune on billion-parameter models with moderate-scale datasets — say, Llama-7B with 10K+ training samples — while keeping end-to-end latency under an hour. That would move this from proof-of-concept to deployable privacy infrastructure for real MLaaS workflows. The obvious experiment not run: applying VeilTune to larger models (GPT-2 XL, Llama-scale) or to tasks requiring more than few-shot data. The honest read is (a) — compute and communication costs of MPC scale harshly, and running even RoBERTa-large in full 2PC is already expensive. The few-shot regime is where the protocol is tractable today. Larger scale is the next paper, not a hidden failure.