Three Small Models Learn Romanized Nepali From 9,000 Examples — But the Metrics Can't Agree on Which One Wins

QLoRA fine-tuning triples chrF++ and cuts perplexity by ~65% for Romanized Nepali, but likelihood and overlap metrics rank models differently — raising questions about what 'best' means for low-resource adaptation.

Imagine three students all take the same crash course in an unstandardized dialect, then sit two different exams. One student aces the essay exam (fluency under the hood) but ties on the multiple-choice (surface overlap with reference answers). The other students reverse the pattern. Who learned the language better? The answer depends entirely on which exam you trust — and that's the core tension this paper surfaces for low-resource NLP. The committed claim: QLoRA fine-tuning on just 9,000 deterministically transliterated instruction examples is sufficient to shift 2-3B parameter models from near-zero Romanized Nepali capability (54-92% of zero-shot outputs were English-like) to functional generation, but the two standard evaluation regimes — reference-overlap metrics (chrF++, BLEU) and byte-normalized likelihood (bits per byte) — disagree on model ranking. This is not a methods paper dressed up as a result. It's a measurement paper that honestly reports a disagreement most teams would quietly paper over. The experimental setup is clean and reproducible: three model families (Llama-3.2-3B, Qwen2.5-3B, Gemma-2-2B), one shared QLoRA recipe, three seeds each, 1,000 held-out test examples. Llama wins on BPB in every single seed — 0.042 lower than Qwen (95% CI: 0.039–0.046). Yet on chrF++ the two are statistically inseparable at 30.51 vs 30.61, even with paired item-level tests. Gemma-2-2B, despite having the lowest tokenizer fertility (2.72 tokens/word, meaning its tokenizer splits Romanized Nepali the least), scores worst on every metric. Lower fertility did not predict better adaptation. This is where the paper gets genuinely interesting for anyone building low-resource pipelines. The conventional wisdom — lower tokenizer fertility means better downstream performance for underrepresented scripts — takes a direct hit. Gemma's tokenizer is the friendliest to Romanized Nepali text by construction, yet the model trails on both likelihood and overlap. The paper doesn't speculate wildly about why, but the implication is clear: tokenizer fertility is a necessary-but-not-sufficient condition, and pre-training data composition likely dominates. The supplementary Llama-3.1-8B comparison is the quiet kicker. Tripling the parameter count (3B → 8B) buys 0.053 BPB improvement but only +0.68 chrF++ and +0.58 BLEU. The likelihood metric sees the bigger model's advantage clearly; the surface-overlap metrics barely register it. If you're deciding whether to spend 3x the compute on the bigger model for a production deployment, the answer depends on which metric your deployment actually cares about — and this paper is honest enough to make that tension explicit rather than picking the metric that flatters the result. Integrity-wise, the multi-seed design with 95% confidence intervals is above the norm for this subfield. The zero-shot baselines under two different prompts acknowledge that prompting strategy matters. The main limitation is that all evaluation is automatic — no human evaluation of output quality, fluency, or factual correctness. For a language with no standardized orthography, the gap between reference-overlap and actual communicative quality could be enormous. The authors know this; they call it an output audit rather than a quality assessment. The broader lesson for practitioners: when you're adapting small models to genuinely low-resource, non-standardized language varieties, don't trust a single metric family. BPB and chrF++ are measuring different things — the model's internal confidence versus its surface similarity to one possible spelling of an answer — and in a no-standard-orthography setting, those things can genuinely diverge. This paper doesn't solve that problem, but it documents it with enough rigor that the next team can't pretend it doesn't exist.