You know how a good mechanic listens to your engine for three seconds and says 'that's the alternator, not the starter'? They're not running diagnostics yet — they're pattern-matching the symptom to the right tool before they open the hood. This paper tries to build that same triage instinct for prompt engineering: a quick numerical check that tells you whether a given knowledge relation should be prompted with a masked template (BERT-style: 'PersonX feels [MASK]') or a prefix template (T5-style: 'PersonX feels → ') — before you burn compute on the wrong one. The committed claim: a metric called the Maskability Index (MI), computed from differences in DepthRank scores between masked and unmasked prompt templates, is positively correlated with downstream generation performance on knowledge base completion tasks. This is not a new model or a new prompting method — it's a diagnostic that predicts which existing method will work better for a given relation type. The mechanism is straightforward. DepthRank is an existing measure of how well a token fits a particular position in a template, based on the depth of that token in the model's probability ranking. MI takes the difference between DepthRank scores under masked templates versus unmasked (prefix) templates. A high MI means the relation's answer tokens rank much better in masked positions — so use BERT-style prompting. A low or negative MI means prefix-style wins. The idea is structurally clean: you're measuring how much the pretraining objective's shape matches the shape of your downstream query. The evaluation runs on ATOMIC2020, a commonsense knowledge base with diverse relation types (physical, social, event-based). The authors show positive correlation between MI and generation quality across these relations. That's the right benchmark — ATOMIC2020 is a standard in commonsense knowledge extraction, with enough relation diversity to stress-test whether MI generalizes across relation types rather than overfitting to one pattern. But here's where you should squint. The entire validation loop is one benchmark, one metric (correlation with generation performance), and no comparison against simpler heuristics. Could you get the same triage accuracy by just looking at template perplexity? Or by running five examples through each mode and picking the winner? The paper doesn't say, because those baselines aren't tested. The correlation is positive, but we don't see effect sizes, and we don't know if MI adds signal beyond what a practitioner would get from ten minutes of trial-and-error. The practical value is clearest in low-resource settings where you genuinely cannot afford trial-and-error across many templates. If you're extracting relational knowledge from pretrained models with limited compute or limited labeled data, MI gives you a principled first guess. That's a real use case — prompt selection is a genuine friction point in few-shot knowledge extraction — but the paper doesn't demonstrate cost savings in wall-clock terms or compare against the obvious cheap baselines. The architecture lineage here runs through the LAMA probing work (Petroni et al.), AutoPrompt, and the broader prompt-engineering literature that treats template selection as a first-class optimization problem. MI sits in the 'meta-prompting' branch — tools that help you pick prompts rather than tools that generate better prompts. It's a modest but useful niche, and the DepthRank foundation gives it more structure than ad-hoc heuristics. The successor question is obvious: does MI predict performance on benchmarks beyond ATOMIC2020, and does it work for models beyond T5 and BERT? The authors don't run those experiments, and the honest read is (a) — scope and compute constraints on a two-author paper, not suppressed negative results.
