Imagine you ask a parrot that has memorized thousands of weather reports to tell you the chance of rain. It will say "70%" — not because it checked the barometer, but because "70%" is the kind of thing that comes after "chance of rain" in the sentences it learned. The number sounds precise. It means nothing. This is exactly what happens when you ask an LLM for a confidence score, and this post by an unnamed practitioner lays out the problem with unusual clarity, then does the harder thing: builds the replacement. The core claim is simple and correct. An LLM's stated confidence is a token-prediction output, not a measurement. When the model writes "0.85," it is selecting the next plausible token in the sequence, shaped by everything preceding it in the conversation — including its own prior sentences. If the model has already written "This is clearly the right answer," a high confidence number is the natural continuation of that text, not an independent assessment of evidence. The author demonstrates this with three falsification tests anyone can run: ask the same question twice and get different numbers, rephrase the question and get different numbers, or note how the number follows the tone of the preceding text rather than the underlying evidence. The post then does something most critique pieces skip — it defines what a trustworthy confidence score actually requires. Two properties: the score must reflect the amount of information behind the prediction, and it must be calibrated against outcomes on the same kind of question. The weather forecaster analogy is load-bearing here. A forecaster who says "70% rain" is calibrated if, across all their 70% days, it rained about 70% of the time. This is verifiable. An LLM's "0.85" has never been verified against the specific task you're asking about. The model's pretraining calibrates it on next-token prediction, not on whether its confidence numbers match its actual accuracy on your customer-support questions or your code reviews. The practical framework the author builds is textbook Bayesian calibration applied to agent outputs. You start with a base rate from historical data, collect observable features about each prediction without asking the LLM how it "feels" (did the search return relevant articles? do repeated queries produce consistent answers?), log every prediction alongside its outcome, and then build a lookup: replies with feature-profile X were correct Y% of the time. This is unglamorous, requires human review to establish ground truth, and works. It has the two properties the LLM's number lacks: same inputs always yield the same confidence, and the score is calibrated on your actual task. The sharpest insight is also the most uncomfortable one for agent builders: without a record of predictions checked against outcomes, there is no confidence score — period. Not a bad one, not an approximate one. None. Most teams shipping LLM agents skip this record-keeping entirely, which means they are flying on numbers that have never been measured. The post doesn't say this will always fail catastrophically. It says you have no way to know whether it's failing until you do the measurement work. What the post does not address is the growing body of work on extracting calibrated probabilities from LLM token-level logits rather than from generated text — methods like verbalized calibration prompting, or using the actual softmax distribution over answer tokens as a proxy for uncertainty. These approaches sidestep the autoregressive text-generation problem by looking at the model's internal probability distribution rather than its written output. They are not perfect, and they have their own calibration issues, but they represent a middle ground between "ask the LLM and trust the number" and "build an entirely external scoring system." The post's framework remains correct as the gold standard; the logit-based approaches are a faster approximation that the author chose not to cover. The piece is a blog post, not a peer-reviewed paper, and it reads like one — clear, well-structured, and aimed at practitioners who are building agents today and need to understand why the shortcut doesn't work. It introduces no new empirical results. Its contribution is pedagogical: making the calibration argument accessible to people who may never have encountered it formally, and providing a concrete five-step recipe they can implement. For that audience, it succeeds completely.
