Imagine you're a restaurant host deciding whether to seat guests yourself or call the manager. Seating them yourself is free when you get it right, costs a short delay when you punt to the manager, but costs five times that if you seat a party of eight at a two-top. The smart move isn't always routing — it's knowing when you don't know. This paper applies exactly that logic to IT ticket classification, and the punchline is brutal for LLM enthusiasts: a 1990s-era TF-IDF pipeline knows when to ask for help far better than GPT-4o-mini does. The committed claim: cost-aware abstention — letting a classifier say 'I'm not sure, send this to a human' — is a better evaluation framework for IT ticket routing than raw accuracy, and classical ML models exploit it far more effectively than prompted LLMs. On a public 47,837-ticket, eight-class corpus, TF-IDF with logistic regression and a calibration-selected threshold of 0.60 reduces expected cost from 0.763 to 0.472 under the paper's cost model (free correct routes, cost-1 human triage, cost-5 wrong routes). That's a 38.1% relative cost reduction with wrong routes dropping from 15.3% to 3.5%. The frozen calibration-selected policies beat both always-route and always-defer baselines in 14 of 16 settings. The ladder here is damning for LLMs. Fine-tuned DistilBERT hits 0.879 accuracy on a stratified 1,000-ticket test subset. The strongest prompted LLM configuration manages 0.477 — barely above chance on eight classes. But the real kill shot is calibration quality: GPT-4o-mini few-shot produces a single confidence value for all predictions, yielding a correctness AUROC of 0.500. That's literally a coin flip. It cannot distinguish its own correct answers from incorrect ones, which means cost-aware abstention has nothing to work with. Architecturally, this sits in the selective-prediction / reject-option family — a classical decision-theoretic technique where you set a confidence threshold and defer uncertain cases. The paper's contribution is methodological framing, not algorithmic novelty: freeze a threshold on calibration data under explicit cost parameters, then evaluate once on held-out test data. The non-LLM classifiers (TF-IDF + LogReg, LinearSVC, Random Forest, DistilBERT) produce well-separated probability distributions that isotonic calibration can sharpen. The LLMs produce degenerate confidence distributions — often a single value repeated across all predictions — making threshold-based deferral meaningless. Integrity is above average for an applied ML paper. The evaluation uses a public dataset, reports bootstrap confidence intervals, includes a post-hoc cost-parameter sweep across 16 settings, and explicitly scopes its conclusions to this dataset, prompt design, and cost model. The authors state outright this is an evaluation framework, not a deployment certification. The honest weakness: the cost model is illustrative (0, 1, 5), not derived from real operational data, and the LLM evaluation uses a 1,000-ticket stratified subset rather than the full corpus — a compute-budget constraint the authors acknowledge. The broader significance is methodological. Most LLM deployment papers evaluate accuracy and call it done. This paper asks the harder question: does the model know when it's wrong? For IT operations teams evaluating whether to replace human triage with LLM routing, the framework provides a concrete, reproducible way to answer that question under their own cost parameters. The finding that seven of eight prompted LLM configurations defer 100% of tickets at the analytically derived break-even threshold of 0.80 is the kind of result that should appear in every enterprise AI procurement discussion. The successor experiment the authors didn't run is fine-tuning an LLM on this corpus rather than prompting it zero- or few-shot. The honest read: this is almost certainly a compute-budget constraint (single author, conference paper), not a strategic omission. A fine-tuned Llama-3 or GPT-4o-mini would likely close much of the accuracy gap with DistilBERT and produce better-calibrated confidence scores. Whether it would justify the cost premium over TF-IDF + LogReg is exactly the question the framework is designed to answer.
