Response-Only LoRA Still Beats Reasoning Traces for Domain Adaptation — And That's the Interesting Finding

Ablation study on Qwen3-8B finds that adding teacher reasoning traces to LoRA fine-tuning doesn't help in-domain finance tasks and may hurt. Simpler wins.

Imagine you're training a new barista. You could hand them a recipe card (response-only), or you could have a master barista narrate their entire thought process while making each drink (reasoning traces). Intuitively, the narration should help — the trainee sees the 'why' behind each step. This paper tests that intuition on LLMs and finds the recipe card wins. The committed claim: for LoRA-based domain adaptation of Qwen3-8B on financial NLP tasks, including response labels alone in the training loss outperforms including teacher-model reasoning traces, whether via a single adapter or a dual-adapter composite setup. The reasoning traces — generated by a stronger teacher model and included in the supervised loss — actually degrade in-domain calibration and argmax accuracy relative to the simpler response-only regime. General capabilities (MMLU-Pro and other retention benchmarks) stay roughly intact across all three regimes, which is the good news, but the reasoning traces buy you nothing extra on the financial tasks. Architecturally, this is standard parameter-efficient fine-tuning territory: LoRA adapters on a frozen Qwen3-8B backbone, with the experimental variable being what gets included in the supervised loss signal. The three regimes — response-only, single-adapter reasoning (traces + labels in one adapter's loss), and dual-adapter composite (separate adapters for traces and labels) — test whether chain-of-thought distillation from a teacher helps downstream task performance when the adaptation is low-rank and domain-specific. The dual-adapter setup is the most architecturally novel idea here, but it yields no further gain over single-adapter reasoning, which itself trails response-only. The integrity picture is mixed. This is an ablation study, which means the paper is grading its own homework across multiple configurations of the same system. The benchmarks — financial NLP tasks for in-domain, MMLU-Pro for retention — are reasonable choices, but the paper doesn't name specific SOTA systems on these financial benchmarks or compare against full fine-tuning or other PEFT methods like QLoRA or adapter-tuning variants. The 'extensive set of ablations' framing is honest about what this is: a systematic exploration of a design space, not a claim of beating any external baseline. The trace-compression experiment (summarizing teacher traces before training) adds a useful data point showing that shorter summaries shift calibration toward the response-only regime without matching it. The milestone question is where this gets practically interesting. The finding that reasoning traces don't help LoRA domain adaptation pushes back against the growing assumption that chain-of-thought distillation is universally beneficial. The next concrete number to watch: at what LoRA rank or model scale do reasoning traces start to help? If traces only help at rank-64 or above, or only on 70B+ models, that's a meaningful boundary for practitioners choosing fine-tuning recipes. The obvious experiment not run: full fine-tuning comparison. Reasoning traces might help when you're updating all parameters but hurt when you're constrained to a low-rank subspace — the adapter may simply lack the capacity to absorb both the reasoning pattern and the task-specific mapping simultaneously. The authors likely didn't run this because compute cost scales dramatically with full fine-tuning on an 8B model, and the paper's scope is explicitly LoRA. But it's the experiment that would tell us whether the finding is about reasoning traces generally or about the interaction between traces and low-rank adaptation specifically. For practitioners doing domain adaptation on financial text today, the takeaway is concrete and actionable: skip the reasoning traces, train response-only LoRA, and spend your compute budget on data quality instead of trace generation. The finding aligns with a broader pattern in the field where simpler supervision signals often outperform elaborate ones when the adaptation budget is constrained.