A Stress Test for AI Decision-Making Exposes How Models Collapse When Options Get Tangled

New benchmark with 5,448 questions across 16 task types reveals that decision models handle trivia fine but crater on interdependent options, constrained choices, and multi-select tasks.

Imagine you're at a restaurant with a fixed-price menu: pick one appetizer, one main, one dessert. Easy. Now imagine the menu has 1,024 items, some combinations are banned by dietary restrictions, several items share ingredients that conflict, and you need to pick the best three simultaneously rather than one at a time. That's the gap MMDM is designed to measure — not whether AI models know things, but whether they can make structured choices when the option space gets combinatorially messy. The committed claim: existing benchmarks for System-1 decision models (models that output a probability distribution over options in a single forward pass, no chain-of-thought) are contaminated by training-set leakage and don't stress-test the dimensions that matter for real deployment — long contexts, large option spaces, and logical dependencies between choices. MMDM fills that gap with 5,448 fresh questions across 16 categories, with option spaces ranging from 2 to 1,024 and inputs up to 64,817 tokens. The ladder results are where this gets interesting. Perplexity's Decider-V1.2, the top-performing model on world-knowledge questions, hits 78.75% Hard Single-select Accuracy on common knowledge — respectable. But drop it into option-logic tasks and it falls to 53.95%. Constraint planning? 47.95%. And the best Hard Multi-select Accuracy across all 28 models tested is a dismal 36.81%. The pattern is consistent: models that look competent on factual recall disintegrate when choices become interdependent. This is not a scaling problem. It's an architectural one — current single-pass decision heads don't have a mechanism for joint option reasoning. Architecturally, this paper evaluates System-1 decision models, which sit in the family of discriminative classifiers applied to LLM-encoded representations. The distinction from System-2 (autoregressive reasoning chains) is load-bearing: System-1 models trade reasoning depth for latency, making them candidates for real-time agent pipelines where you can't afford chain-of-thought. The benchmark itself is model-agnostic but implicitly tests whether the softmax-over-options paradigm can handle combinatorial structure. The integrity regime is mixed. On the positive side, the benchmark is freshly constructed with source-grounded recompositions designed to defeat memorization, 28 models are evaluated (good breadth), and the project page is public. On the negative side, there's no pre-registration, no independent replication, and the authors designed both the benchmark and the evaluation — a classic grader-grading-own-homework situation. The newly constructed tasks haven't been validated by a third party for difficulty calibration or bias. The milestone this sets up is concrete: the 36.81% ceiling on Hard Multi-select Accuracy is the number to watch. Any model that cracks 60% on MMDM multi-select while maintaining sub-100ms latency would be genuinely deployment-ready for agent tool-selection and constrained planning. We're roughly one architectural innovation away — probably something that lets the decision head reason about option-option interactions without full autoregressive generation. The obvious experiment not run: fine-tuning a System-1 model specifically on MMDM-style tasks and measuring whether the failure modes are learnable or structural. The authors benchmark 28 off-the-shelf models but never ask whether targeted training closes the gap. My read: they're saving it for the follow-up paper, because a benchmark paper that also proposes a fix is two papers stapled together.