No LLM Required: Feature-Engineered Table Retrieval Matches Claude-Powered Systems at a Fraction of the Cost

FERN replaces generative LLM calls with frozen retrievers and a LightGBM ranker, matching or nearly matching RASL's table recall on Spider and BIRD at 208ms per question on a single GPU.

Imagine you're organizing a massive library. One approach hires an expensive expert to read every book title and decide which shelves to check. The other approach runs three cheap keyword searches, pulls 100 candidate books, then lets a trained librarian quickly sort the pile by 120 observable features — spine thickness, publication year, shelf proximity, call-number overlap. FERN is the second librarian. It never reads the books; it just ranks the pile fast. The committed claim: you can replace generative LLM-based table selection (specifically RASL's Claude 3.5 pipeline) with a feature-engineered LightGBM ranker over frozen retriever candidates, and match or nearly match recall on cross-benchmark pooled catalogs — without any hosted model calls. This is not a new retrieval paradigm; it is an engineering argument that the expensive part of the pipeline (the LLM) is doing less work than people assume. FERN fuses three frozen retrievers to produce 100 candidate tables per question, then computes up to 120 features per candidate — retrieval scores, schema overlap signals, metadata features — and feeds them into LightGBM rankers trained once on BIRD's training split. The critical test is generalization: on Spider (never seen during fitting), the candidate pool alone hits RASL's published recall@15 of 98.0, and the ranker pushes complete@5 from 83.9% to 95.6%. At recall@5, FERN lands within one point of RASL on Spider (96.7 vs 97.0) and on BIRD-with-evidence (96.6 vs 97.5). The gap widens to 8.2 points on BIRD without evidence hints (89.3 vs 97.5) and the system struggles on Fiben at shallow cutoffs (54.2 vs 69.2 at recall@15), though it closes at deeper retrieval (80.7 vs 79.4 at recall@50). The ladder positioning is honest but uneven. RASL is the named baseline throughout, and the authors report both wins and losses without hiding the gaps. On BEAVER — 121 real-world questions from a different distribution — FERN's recall@15 (82.1) falls between two BEAVER-trained retrievers (80.8 and 84.5), but its complete@15 is below both (49.6 vs 54.5 and 62.0). This is the kind of candor you want: the system generalizes respectably but doesn't pretend to dominate everywhere. The Fiben shortfall is the loudest signal that FERN's feature engineering hasn't fully captured whatever structural signal the LLM extracts from complex schema relationships. The integrity picture is mixed. The cross-benchmark evaluation design is strong — train on BIRD, test on Spider, BEAVER, Fiben — which controls for overfitting to a single catalog. But there's no code release mentioned, no pre-registration, and the RASL comparison uses published numbers rather than a controlled re-run. The feature ablation on BIRD dev (non-retrieval features add 7.7 points to complete@5) is the strongest internal validation, showing the ranker is learning real signal beyond retrieval scores. The practical upshot is cost and latency. Median 208ms per question on one A100, no API calls, no hosted model dependency. For teams building text-to-SQL over enterprise catalogs where latency and cost matter, FERN says: you can get 95%+ of the LLM-powered recall with a LightGBM model and frozen embeddings. The remaining 5% gap is real, and for hard cases (BIRD without evidence, Fiben's complex schemas) the gap is larger. The obvious next experiment is end-to-end text-to-SQL accuracy — does the 1-8 point recall gap actually matter downstream when the SQL generator sees 5 or 15 tables? The authors benchmark retrieval in isolation. The honest read: this is likely being saved for a follow-up paper, because if end-to-end accuracy is comparable, the argument becomes much stronger, and if it isn't, the paper's value proposition weakens.