LLMs Don't Plan Ahead — They Re-Read the Instructions and Wing It, and Researchers Found the Exact Neurons That Do It

Mechanistic study finds LLMs encode sub-goals in a sparse set of attention heads, but sustain them not through internal recurrence — by repeatedly re-reading the prompt.

You know that person who cooks a complex recipe not by memorizing all the steps but by glancing back at the cookbook every thirty seconds? They check what's done, look at what's next, and execute one step at a time. That's what this paper says large language models are doing when they write code — and the researchers found the exact neural circuitry responsible. The core claim: LLMs encode sub-goals — discrete, human-readable intentions like 'I'm checking if a word is a palindrome' — in the outputs of a sparse set of attention heads, and these encodings causally drive the model's next actions. This isn't a behavioral observation from prompting tricks. The authors localize specific attention heads, decode their activations into natural language descriptions of upcoming sub-goals, and verify causality through interventions. When you transplant a goal encoding from one programming language context into another, the model executes the same operation using the target language's syntax. That's a strong plasticity result — the goal representation is abstract enough to survive environment changes. The most interesting puzzle the paper addresses: how does a transformer, which has no recurrence, sustain a goal over many tokens of execution when it has fewer layers remaining than tokens to generate? The authors' answer, the Non-Recurrent Goal-Seeking (NRGS) account, is elegantly simple. The model doesn't carry forward an internal state. Instead, it re-reads the original task specification from the context, compares it against the visible progress in the growing output, and re-derives what to do next. Every forward pass is a fresh goal-retrieval, not a state-propagation. They test this with targeted causal interventions — modifying the context to make the progress-so-far look different and observing whether the model shifts its sub-goal accordingly. The experimental setup is well-constructed. The authors use a coding task with multiple sub-tasks, which gives them ground-truth decomposition of what the model 'should' be doing at each step. This avoids the usual interpretability trap of projecting human narratives onto activations without a clear behavioral target. The coding domain also provides natural cross-environment tests (same algorithm, different programming language) that would be hard to construct in freeform text generation. Where does this sit in the broader interpretability landscape? The lineage runs through circuit-level mechanistic interpretability (Elhage et al., Conmy et al.) and the emerging sub-field of goal/planning representations in transformers. The key tension in this field is whether LLMs do anything resembling planning or whether their apparent planning is an artifact of pattern completion over training distributions. This paper lands firmly on the 'there are internal goal representations, but they work differently than you'd expect' side. The NRGS account is a genuine contribution — it resolves the recurrence puzzle without invoking hidden state and makes testable predictions. The integrity profile is solid for an interpretability paper. Causal interventions (activation patching, transplantation across languages) are the gold standard for this kind of claim. The main limitation is scope: the experiments are in coding tasks, and the generalization to open-ended natural language generation is undemonstrated. The authors also don't benchmark against competing accounts of goal-like behavior in transformers with quantitative comparisons — the paper is more 'here's a mechanism we found and verified' than 'here's why our account beats the alternatives on metric X.' The practical upshot is significant for anyone building agentic systems. If LLMs sustain goals by re-reading rather than by internal state propagation, then context window management and prompt structure matter even more than you thought — the model's ability to 'plan' degrades with context pollution in ways that internal-state models wouldn't. The sparse attention head localization also opens a path toward goal-level steering: if you know which heads encode sub-goals, you can intervene on them directly rather than through prompt engineering.