Someone Finally Put 'System 1' AI Decision Models on the Stand — Here's What the Audit Found

First independent calibration and explainability audit of single-forward-pass decision models (Jev-style). The 4B+ models are faster and better calibrated than autoregressive baselines, but sub-1B models have a serious cross-talk problem.

You know how a bouncer at a nightclub doesn't reason through your outfit, your ID, and your vibe in sequence — they take one look and decide? That's what System 1 models do for AI decision routing: one forward pass, typed output, no chain-of-thought deliberation. TypeSafe AI's Jev architecture pioneered this approach for low-latency triage, but until now nobody had stress-tested the confidence scores these models emit or asked whether their explanations are faithful. This paper is that stress test. The core claim: System 1 models with 4B+ parameters are faster AND better calibrated than both logprob-based and structured JSON/CoT autoregressive baselines on syntactic perturbation tasks, with weighted Expected Calibration Error (ECE_w) of 0.030–0.087 versus 0.144 for logprobs and 0.378 for JSON decoding. That's not a marginal win — the structured-generation baseline is off by nearly 10× more. The audit spans five open-weight models from 0.8B to 27B parameters across NLI (e-SNLI), adversarial syntax (HANS), synthetic clinical triage (N=900), prompt injection defense (Tensor Trust, N=250), and a bespoke 86-case logic-gate trigger suite. The breadth matters: this isn't a single-benchmark headline. The most technically interesting finding is that provider confidence isn't learned from data — it's a closed-form dispersion transform. For sub-10B models, confidence equals 1 − H(p)/ln K (normalized entropy); for Clef-27B, it's a normalized margin. That means these confidence scores are mathematically guaranteed to track distributional spread, not empirical accuracy. This is good news for interpretability (the score means exactly one thing) and bad news for naive users (it doesn't mean what they think it means — high confidence ≠ high accuracy without external calibration). The cross-question interference result is the paper's sharpest red flag. When you bundle multiple questions into a single request on the smallest model (Tev1-0.8B), 11.7%–22.4% of binary answers flip compared to asking them individually. The 4B model drops this to 0.1%–0.2%. This is a clear scaling threshold: below 4B parameters, System 1 models leak context across decision slots in ways that could silently corrupt routing decisions at production scale. For prompt-injection defense, the ≥4B models hit 0.988–0.998 AUROC with up to 95.2% recall and 0.0% false alarms at 48–60 ms latency on the Tensor Trust benchmark (N=250). Those numbers are strong enough to be deployment-grade for many use cases, though the sample size is modest. The explainability story is equally concrete: single-pass inference makes 512-coalition Shapley value computation interactive at 1.3–3.2 seconds per input, and a KernelSHAP variant with logit link (S1-SHAP) recovers disjunctive OR-gate structure at 89.3%–100% versus 57.1%–82.1% in probability space (p_adj = 0.004). The validation regime is honest but not independent. The author designed the benchmark, ran all experiments, and evaluated the results — no external replication exists. The benchmarks themselves are a mix of community standards (e-SNLI, HANS, Tensor Trust) and custom-built suites (the 86-case logic gates, the 900-case clinical triage). Code and artifacts are released, which invites replication, but nobody has done it yet. The paper is transparent about what the confidence scores are and are not, which is more intellectual honesty than most deployment-facing ML papers manage. The missing experiment is the obvious one: production-scale evaluation under distribution shift with real (not synthetic) clinical or security data. The clinical triage dataset is synthetic (N=900), and the prompt-injection benchmark, while community-sourced, is small (N=250). The paper establishes the measurement framework but doesn't yet answer the question practitioners actually need: do these calibration properties hold when the inputs come from the wild? That's likely a compute-and-data access gap, not a result the authors are hiding.