You know how a spell-checker doesn't just tell you a word is wrong — it tells you how wrong, what letter is off, and whether the fix is stable? Robot simulation benchmarks today are the opposite: they tell you 'task completed' or 'task failed,' full stop. Specola is the spell-checker for robot evaluation — it takes the binary pass/fail and replaces it with a continuous, multi-dimensional signal that says how close the robot was to violating each requirement, whether success was fragile or robust, and whether intermediate constraints held even when the final goal was met. The committed claim: Specola is the first open tool that wraps arbitrary robot simulators in Signal Temporal Logic (STL) monitors, exposing a continuous robustness score for every specification rather than just a binary success flag. STL is a formal language that assigns real-valued 'robustness' to temporal properties — positive means satisfied, negative means violated, and the magnitude tells you by how much. The paper's contribution is not the math (STL monitoring is well-established in the formal methods community) but the engineering bridge: connecting STL monitors to the internal state of nine different simulators and benchmarks, including LIBERO, RoboCasa, Isaac Lab, and RoboDojo. Architecturally, Specola sits as middleware. It provides a Python API and CLI that intercept simulator state at each timestep, map it to specification-relevant signals (position, velocity, contact force, etc.), and feed those signals into STL monitors. The adapter pattern means you write one thin adapter per simulator to expose its internal state, and the rest of the monitoring infrastructure is shared. Nine adapters ship at release. This is a plumbing contribution — not glamorous, but the kind of infrastructure that changes how a field measures itself. The integrity question is straightforward but worth stating plainly: the paper validates Specola via case studies, not via a controlled experiment comparing evaluation regimes head-to-head. The case studies show that Specola can distinguish between policies that have identical success rates but different robustness profiles — e.g., one policy barely clears a threshold while another has wide margin. That's illustrative, not experimental. There's no pre-registration, no ablation, and no user study. The code is publicly released, which is the strongest integrity signal here. Where does this sit on the ladder? The closest prior work is ad-hoc metric augmentation in individual benchmarks (RoboCasa tracks sub-task completion, LIBERO has some partial-credit metrics). No existing tool provides a unified, formal-language-based evaluation layer across simulators. Specola isn't beating a SOTA number — it's proposing a new measurement axis. The question isn't 'is this better than X' but 'does the field adopt this as evaluation infrastructure.' The milestone to watch is adoption, not accuracy. If two or three major robotics groups start reporting STL robustness scores alongside success rates in their next benchmark papers, Specola will have shifted the evaluation norm. If it stays a single-lab tool with nine adapters, it's a well-built hammer waiting for nails. The gap between 'released' and 'adopted' is 1-3 years, and it depends entirely on whether the adapter-writing friction is low enough for busy lab students. The obvious experiment the authors didn't run: a head-to-head study where two policy-selection pipelines — one using success rate, one using STL robustness — lead to different downstream choices, and the STL-guided choice is measurably better in deployment. This would convert 'richer evaluation is possible' into 'richer evaluation is necessary.' Honest read: this is a v1.0 release paper, and that experiment is the obvious follow-up — likely saved for the next paper rather than out of reach.
