Imagine you're a pharmacist and someone hands you a prescription that says 'administer 500mg ibuprofen.' Safe or not? You literally cannot answer without knowing what else the patient is taking. If they're on blood thinners, it's dangerous. If they're not, it's routine. The prescription is identical — the patient state changes the answer. That's the core mechanism of this paper, applied to language models judging commands sent to industrial controllers. The committed claim: LLMs evaluating natural-language commands to industrial equipment (motors, valves, heaters) cannot reliably distinguish safe from unsafe commands when they lack the machine's current state — and standard evaluation methods don't even check whether that information was provided. The same command ('open valve V1') can be safe in one state and catastrophic in another. The paper proposes an 'answerability audit' — a diagnostic that pairs identical commands across opposing states and counts success only when the model gets both right. The test design is deliberately small: 15 command pairs across three industrial systems (motor, tank, heater), evaluated with four LLMs (GPT-4o, Claude 3.5, Gemini 1.5, Llama 3.1 70B). Without state information, no model reliably solved the paired cases. One model always permitted commands; another always denied them. Their raw accuracies differed by 32 percentage points, but both achieved identical 50% balanced accuracy — coin-flip performance when you weight safe and unsafe cases equally. The raw accuracy gap was an artifact of class imbalance, not capability. With explicit state values added to the prompt, three of four models improved, scoring up to 13 out of 15 correct pairs. The exception is notable: one model remained at zero pairs even with state. The authors also tested a deterministic rule evaluator applying the same logic used to generate the expected answers — it matched all 15 pairs, but as the authors candidly note, this is tautological, not an independent safety validation. The architectural insight here isn't about model quality — it's about evaluation methodology. Current benchmarks for LLM safety in industrial contexts tend to measure single-command accuracy without controlling for input completeness. The paired-score metric forces a harder question: does the model actually use state information, or is it pattern-matching surface features of the command text? The answer, at this scale, is sobering: without state, the models default to biases (always-permit or always-deny) that happen to look competent on imbalanced test sets. The paper is upfront about its limitations: 15 pairs is a proof-of-concept, not a reliability study. The three systems are author-constructed, not drawn from operational deployments. The rule evaluator's perfect score is definitionally circular. These are honest disclosures, and the authors never claim general model reliability. What they do claim — that answerability should be audited before accuracy is measured — is both obvious in retrospect and systematically ignored in practice. The practical takeaway is narrow but important: anyone deploying LLMs in industrial safety loops should verify that the model's input pipeline includes real-time machine state, and should evaluate using paired cases that require state-dependent reasoning. The audit is a diagnostic, not a fix — but diagnostics that reveal a category of invisible failure mode earn their keep.
The analogy
You're a pharmacist handed a prescription with no patient chart. 'Administer 500mg ibuprofen' — safe or not? Without knowing the patient's current medications, you can't answer. The prescription is identical; the patient state flips the safety verdict. This paper shows LLMs judging industrial commands face the same problem: same command, different machine state, opposite correct answer — and nobody was checking whether the model even received the state.
Pharmacist
Same prescription requires opposite decisions depending on patient state (current medications, allergies)
LLM safety judge
Same industrial command requires opposite safety verdicts depending on machine state (valve positions, temperatures)
Honest assessment
This is a workshop-stage methodological contribution that identifies a real and overlooked problem: LLMs asked to make safety decisions about industrial commands often aren't given the machine state needed to decide, and standard accuracy metrics hide this failure. The 15-pair proof-of-concept is too small for reliability claims but cleanly demonstrates the diagnostic principle. The balanced-accuracy analysis — showing how two models with 0 correct pairs can appear 32 percentage points apart on raw accuracy — is the most immediately useful analytical contribution. This matters to anyone evaluating LLMs for industrial safety roles, but needs operational-scale replication before it changes practice.
What Is CLAIMS?
50
/ 100
BedrockNews applies a structured analytical framework called CLAIMS to academic papers and technical research releases.
How CLAIMS worksClaim
3/5
Ladder
2/5
Architecture
2/5
Integrity
3/5
Milestone
2/5
Successor
3/5
What is this paper actually committing to?
Methodological, not capability. The claim — that answerability should be audited before accuracy — is sound and overlooked, but it's a diagnostic framework, not a breakthrough result. The 15-pair proof-of-concept establishes the idea.
The field fight
Can LLMs serve as safety-critical decision layers for industrial control systems, or do they require structured verification systems around them?
Side A
LLMs can reason about safety with sufficient prompting and fine-tuning
Held by: Various NLP-for-ICS papers, Microsoft Industrial Copilot framing
Side B
LLMs need explicit state input pipelines and structured validation; prompting alone is insufficient for safety-critical decisions
Held by: This paper, formal verification community, ISA/IEC 62443 safety standards
Milestone tracker
Unlocks: Evidence-based guidance on whether LLM safety layers are viable in deployed industrial control systems
Current
15
command pairs
this paper (synthetic)
Next
100+
command pairs
operational industrial data
Requires industrial partnership for real SCADA/PLC command logs; timeline unclear
Verified Facts (7)
15 pairs from three systems with four language models tested
None reliably distinguished paired cases without state, even with rules
With state, scores ranged from 0 to 13 correct pairs out of 15; three models improved
One model always allowed commands and another always denied them
Overall raw accuracies differed by 32 percentage points; both scored 50% balanced accuracy and no correct pairs
Rule evaluator matched all 15 pairs by applying the same rules that supplied the expected answers
8 pages, accepted for IEEE ICDMW 2026, 6th MLC Workshop
Practical Implications (4)
Anyone deploying LLMs in industrial safety loops should verify the model's input pipeline includes real-time machine state, not just command text
Evaluation of LLM safety judges should use paired-score metrics (same command, opposing states) rather than single-command accuracy
Raw accuracy on imbalanced safety datasets is misleading — balanced accuracy and paired scoring expose default-permit or default-deny biases
The answerability audit is a pre-check before model evaluation: does the model even have the information needed to decide?
More latest news
Read all