Imagine you're a hospital administrator evaluating surgeons. You give each one a single score: "operations completed." Surgeon A scores 95%. Great — hire them. But what if Surgeon A is flawless at appendectomies and silently catastrophic at anything requiring fine motor control? The aggregate number hid a lethally uneven ability profile. That's exactly what's happening when we compress LLMs and evaluate them on task benchmarks. ACE-Compression's committed claim: standard compression evaluation is structurally blind. Task-level accuracy ("did it answer correctly?") collapses a multi-dimensional ability profile into a scalar, and that scalar hides which abilities survived and which quietly broke. The framework replaces this with a four-dimensional probe — procedural reasoning and constraint execution, knowledge grounding and truthfulness, social cognition and affective understanding, and persona fidelity and stylistic consistency — applied across 35+ compressed checkpoints from six open-weight LLMs using both quantization and pruning. The central finding is that compression doesn't uniformly degrade performance — it redistributes the ability profile. Coarse, local, surface-level signals (basic factual recall, simple instruction following) persist robustly. Fine-grained, long-tail, and context-dependent abilities (multi-step constraint satisfaction, nuanced persona maintenance, affective reasoning) become fragile first. This is not a surprise if you think about it mechanically — compression removes information, and the information that's most compressible (high-frequency, widely distributed) survives, while the information that's rare and distributed across specific weight configurations doesn't. On the ladder: quantization consistently outperforms pruning in ability retention across all four dimensions, confirming what the community has broadly suspected but rarely measured at this granularity. The paper doesn't claim to beat any compression method — it's a diagnostic framework, not an algorithm. The real contribution is giving deployers a way to match compression strategy to application needs. If your chatbot needs persona fidelity, the framework tells you which compression scheme destroys that ability and which preserves it. If you need constraint execution for code generation, different answer. Architecturally, this is a benchmark-and-evaluation paper, not a model paper. The six base models span the current open-weight landscape (the paper covers quantization via GPTQ/AWQ-style methods and structured/unstructured pruning). The evaluation probes are designed as targeted ability tests rather than general benchmarks — closer to neuropsychological testing than to a standardized exam. The framework itself is lightweight: the cost is in running the probes, not in any novel training or inference procedure. Integrity is mixed. The authors test across 35+ checkpoints and six models, which provides breadth, but all evaluation is self-designed — there's no external validation of whether the four ability dimensions actually capture what matters for deployment. The probes are reasonable but not pre-registered, and there's no independent replication. Code is released, which is the strongest integrity signal here. The choice of four dimensions (rather than three or seven) isn't derived from theory; it's a practical design choice the authors defend but don't prove is exhaustive. The practical takeaway is actionable today: if you're deploying a compressed LLM, stop looking only at aggregate benchmark scores. Profile the specific abilities your application needs and test those directly. ACE-Compression provides one such profiling framework — whether it's the right four dimensions for your use case is an open question, but the diagnostic instinct is sound and overdue.
