Imagine you're a wine sommelier judging whether a student's tasting notes capture what matters about a Burgundy. You wouldn't count matching words — you'd ask: did they notice the terroir, the tannin structure, the way acidity frames the fruit? That's the gap this paper targets. Standard NLG metrics like BLEU and BERTScore count lexical overlap or embed whole texts into single vectors, but neither asks whether two texts make the same things salient — the same connotations, cultural references, discourse-level framings. This paper calls that layer the 'semiotic profile' and builds a framework to measure alignment at that level. The committed claim: existing NLG evaluation metrics miss a meaningful layer of textual meaning — the semiotic layer, where context-dependent connotations and discourse references live — and two new scores (Semiotic Fidelity and Semiotic Coverage) can capture discrepancies that BLEU, ROUGE, and BERTScore cannot. This is a measurement-framework paper, not a model paper. It does not improve generation; it proposes a new yardstick. The architecture sits in the NLP evaluation family, specifically the reference-based metric design space. The approach constructs a 'semiotic profile' for each text — essentially a structured representation of its contextual meanings and discourse references — then compares profiles pairwise. Fidelity measures how much of the reference profile is supported by the generated text; Coverage measures how much of the generated text's profile is recoverable from the reference. The asymmetry is the key insight: a generated text can be faithful to a reference without covering everything the reference says, and vice versa. The authors use LLMs at varying sampling temperatures as their generation source, which is clever — temperature gives them a natural dial to vary the diversity-fidelity tradeoff. The central empirical finding is that coverage is systematically lower than fidelity across experiments, and that alignment between LLM outputs and human-curated data peaks at low sampling temperatures while degrading at higher temperatures. This is intuitive — low temperature means conservative, reference-hugging generation — but having a metric that quantifies this semiotic drift is new. The question is whether the semiotic profile itself is well-defined enough to be robust across domains, annotators, and languages. Integrity is the weak link. The paper compares LLM outputs against human-curated data, but the semiotic profiles themselves appear to be constructed using methods that likely involve subjective annotation or LLM-based extraction — which means the ruler is partly built from the same material it's measuring. There's no mention of pre-registration, and the experimental setup (varying temperature on LLMs, comparing to curated data) is more demonstration than adversarial stress test. The paper does not name a specific competing evaluation metric and show head-to-head performance on a diagnostic task where the semiotic layer is provably the differentiator. The practical milestone here is adoption. Evaluation metrics live or die by whether the community uses them. BERTScore took off because it was easy to compute and correlated better with human judgment than BLEU on standard benchmarks. For Semiotic Fidelity and Coverage to matter, someone needs to show that on a concrete benchmark (say, summarization or style transfer), these scores predict human preference better than BERTScore or GPT-4-as-judge. That experiment is conspicuously absent. The obvious next experiment is a human-judgment correlation study: take 500 generated summaries, collect human quality ratings, and show that Semiotic Fidelity/Coverage correlate with human ratings better than BLEU, ROUGE, BERTScore, and GPT-4-judge. The honest read on why it wasn't run: this is likely a conceptual-framework paper establishing the idea, with the empirical validation planned for a follow-up. That's a legitimate staging strategy, but it means the current paper is a promissory note.
