Imagine you're judging a jazz improvisation. You could compare the final recording to a reference performance (did they hit the same notes?), or you could track how the musician builds tension and release across the solo — does the arc feel human? This paper argues we should do both when evaluating open-ended text generation, and then honestly reports which approach actually tells us something. The committed claim: a reference-based framework that evaluates LLM continuations through three lenses — trajectory alignment (does the model's coherence/diversity evolve the way a human's does over time?), summary comparison (do aggregate statistics match a human continuation of the same prompt?), and reference likelihood (how probable is the model's text under a distribution fit to human writing?). The paper finds that diversity-based trajectory alignment and mean-based summary comparisons both capture quality-related variation in human ratings, but — critically — temporal alignment does not beat simpler baselines. That honesty is the most interesting result. The architecture is straightforward NLG evaluation methodology, not a new model. The framework sits in the reference-based evaluation family alongside BERTScore, MAUVE, and BLEU, but adds a temporal dimension by tracking how coherence and diversity metrics evolve token-by-token through a continuation. Coherence is measured via embedding similarity, diversity through lexical and semantic spread. The 'reference distribution' component uses kernel density estimation or similar fits to human text statistics. No novel neural components — this is measurement infrastructure, not generation. The ladder position is honest but modest. The paper doesn't claim to beat existing automatic metrics on correlation with human judgments. Instead, it offers a structured decomposition — asking which facets of the human-machine gap (trajectory shape vs. summary statistics vs. likelihood) actually matter. The implicit baseline is 'just compare means,' and the paper concedes that temporal alignment doesn't clearly outperform it. Against MAUVE and BERTScore as established reference-based metrics, this framework provides finer-grained diagnostics rather than a single better number. Integrity is mixed. The framework is validated against human quality ratings, which is the right move, but we don't know the scale of the human evaluation (number of annotators, prompts, models tested). The abstract is transparent about the negative result — temporal alignment doesn't beat simpler baselines — which is a good sign. Code is released on GitHub, which is strong. But without knowing whether benchmarks were chosen before or after seeing results, and without independent replication, the integrity story is incomplete. The milestone question for NLG evaluation is: when does an automatic metric correlate with human judgment well enough to replace human evaluation in practice? Correlation thresholds around r=0.7-0.8 with human ratings are typically considered the bar. This paper doesn't report clearing that bar; it reports 'positive associations.' The gap between 'captures quality-related variation' and 'reliably predicts quality' is where the next concrete advance needs to land. The obvious experiment not run: applying this framework to a head-to-head comparison across multiple LLMs of different capability tiers (GPT-4 vs. Llama-3 vs. smaller models) to see if the framework's measures rank models in the same order humans would. My read: this is likely (a) — a single-author paper with limited compute/annotation budget, saving the large-scale model comparison for follow-up work or collaboration.
