Imagine you hand someone a blueprint of a house and ask them to describe it in words. They nail it — room layout, load-bearing walls, plumbing runs, all correct. Now you hand them a photo of that same house and ask them to draw the blueprint. They get the rough shape but miss half the structural details. That asymmetry — fluent description without structural comprehension — is exactly what this paper catches LLMs doing with formal meaning representations. The committed claim: large language models (Claude Sonnet 4.5 and Claude Opus 5) can generate English sentences from Minimal Recursion Semantics graphs at near-expert levels with only three examples, but fail dramatically when asked to parse those same sentences back into MRS. Opus hits 76.3 BLEU on generation — ten points above a seq2seq model trained on 72K sentence pairs — while scoring only 65.5 F₁ on parsing against ACE's 91.0 F₁, with exact-match at roughly 1%. The generation-parsing gap is the result, not either score alone. The ladder here is unusually well-constructed. The baselines are not straw men: ACE is the actual hand-built English Resource Grammar processor, and the trained seq2seq systems from Hajdik et al. (2019) were purpose-built for this task on 72K–1M+ training pairs. The LLMs see three examples. That Opus matches a system trained on a million pairs at generation (76.3 vs 77.2 BLEU) while getting obliterated at parsing (65.5 vs 91.0 F₁) is a clean, interpretable contrast. The pooled-judge variant that lets the model select among ACE's candidate sentences pushes generation to 77.0 BLEU, showing the models can recognize good outputs even when they can't produce the underlying structure. Architecturally, this is a zero-shot prompting study on closed-weight autoregressive transformers (Claude family), tested against both a classical symbolic parser (ACE + ERG) and trained neural seq2seq models. The method leans entirely on in-context learning — no fine-tuning, no task-specific training. The test set is a reconstructed 10K-sentence split from the ERG's Redwoods treebank. The formal representation being tested, MRS, is a graph of predicates and arguments — not free-form text — which makes the parsing direction genuinely hard because the model must produce exact graph structure, not approximate paraphrases. Integrity is solid for what it is: the benchmark is reconstructed from a published treebank split, the baselines are named with numbers, and the evaluation metrics (BLEU for generation, F₁ on predicates/arguments plus exact-match for parsing) are standard. The failure mode analysis for parsing is characterized rather than hand-waved. The main caveat is that this tests two models from one family (Anthropic's Claude) — we don't know if GPT-4o or Gemini show the same asymmetry. The ~1% exact-match on parsing is a striking number that's hard to cherry-pick into looking good. The real contribution is conceptual rather than technical: showing that generation benchmarks can dramatically overstate a model's understanding of formal representations. A model that writes perfect English from an MRS graph looks like it 'understands' MRS. But if it can't produce the MRS from the same English, that understanding is illusory — it's pattern-matching the surface form, not internalizing the graph structure. This is a clean empirical demonstration of a point the formal semantics community has argued theoretically for years. The obvious next experiments: test open-weight models (Llama, Mistral) to see if the asymmetry is architecture-general; fine-tune on even small amounts of MRS parsing data to measure how quickly the gap closes; and test whether chain-of-thought or structured output prompting (asking the model to build the graph incrementally) narrows the parsing deficit. The authors likely didn't run these because the paper's point is about off-the-shelf LLM capability, not about closing the gap — but the fine-tuning experiment would tell us whether this is a fundamental limitation or a training-data gap.
