Your Security Scanner Is Only as Good as the Language You Wrote In — And Nobody's Measuring That Gap

First cross-language SAST tool comparison using real-world EV charging protocol code. Rust: 2 issues. Java: 1,689. The gap is mostly tooling maturity, not code quality.

Imagine you're a food inspector with a different checklist for every restaurant — one checklist has 200 items, another has 5. At the end of the day, you report that Restaurant A has 40 violations and Restaurant B has 1. The headline reads 'Restaurant B is safer!' But you didn't actually measure safety — you measured how thorough your clipboard was. That's the core mechanism of this paper: SAST tools don't measure code quality uniformly across languages; they measure the maturity of their own rule sets per language, and the gap is enormous. The committed claim: this is the first empirical study that takes SAST tools (SonarQube, Semgrep, plus language-specific analyzers) and runs them across twelve implementations of the same protocol spec — OCPP 1.6, the dominant standard for EV charging station management — each written in a different programming language. The unit of comparison isn't 'which codebase is better' but 'how differently do tools behave when the specification is held constant and only the language changes.' That's a genuinely useful experimental design. The numbers are stark. Java's implementation triggered 1,689 total findings; Rust's triggered 2. But the authors are honest about the interpretation: mature languages like Java have decades of accumulated SAST rules, while newer languages like Rust have sparse coverage. The finding count is confounded by tooling depth, not just code quality. SonarQube and Semgrep, the two cross-language tools tested, showed highly uneven rule coverage across their supported languages. Language-specific tools sometimes found issues the generalists missed, and vice versa. The architecture here is straightforward empirical measurement, not algorithmic innovation. Twelve open-source OCPP 1.6 implementations were selected — one per language — and subjected to both multi-language SAST tools and the best available language-specific analyzer. The study design leans on OCPP as a natural experiment: same spec, different implementations, different ecosystems. It's a measurement paper, and the measurement is the contribution. Integrity is mixed. The corpus is real-world open-source code, which is good — no synthetic benchmarks or toy examples. But each language is represented by exactly one implementation, so you can't distinguish language-ecosystem effects from codebase-quality effects. The authors acknowledge this. There's no blinding or pre-registration, but for an empirical SE study at this scale, that's standard. The bigger limitation is that SAST findings aren't ground-truthed against actual exploitable vulnerabilities — you're counting what the tools report, not what's actually dangerous. The practical upshot is pointed: if you're choosing a language for a security-critical system like EVCS software, you should factor in SAST tooling maturity, not just language features. A language with immature tooling creates a false sense of security — fewer reported issues doesn't mean fewer real vulnerabilities. For the EV charging ecosystem specifically, OCPP's spec-level ambiguity means implementation divergence is inevitable, and tooling gaps mean some of that divergence goes undetected. The obvious next step the authors didn't take: manual vulnerability triage. They counted tool findings but didn't classify which findings represent real, exploitable security flaws versus noise. That's the experiment that would make this work load-bearing for policy. My read: it's a resource and expertise constraint — manual triage across twelve languages requires twelve domain experts, and this is a six-page conference paper from what appears to be a university group. The follow-up paper likely exists in someone's head already.