LLMs Can't Reliably Impersonate You on a Personality Test — Even With Your Social Media Posts

Four LLMs tried to answer Big Five questionnaires as specific people. User text didn't consistently help. Models regress to the mean, failing at personality extremes.

Imagine you hand a friend your diary and ask them to fill out a personality quiz pretending to be you. A good friend — someone who's actually read the entries and knows you — should get closer than a stranger guessing from population averages. This paper asks whether LLMs are that good friend, and the answer is: not really. Given social-media posts, four models (GPT-4, Gemma, Llama 3.1, Mistral) completed the BFI-2-60 Big Five inventory as specific users. The predicted trait scores were then compared to the users' actual self-reported scores from the MyPersonality dataset. The core finding is deflating for the "LLM as digital twin" narrative. Providing user text does not consistently improve prediction accuracy over a no-text baseline. Models cluster their answers near the middle of the trait distribution — they're decent at predicting average people and bad at predicting anyone who scores unusually high or low on a trait. This is the regression-to-the-mean problem that haunts every attempt to personalize LLM behavior: the models have learned what a typical human looks like, and they default to that prior even when given individuating information. The ladder here matters. The strongest baseline isn't another LLM — it's population-level statistics. If you just predict every person's score as the population mean, you already do surprisingly well for the middle 60% of people. The paper honestly shows that LLMs often fail to beat this trivial baseline for extreme scorers, which is precisely where a "digital twin" would need to shine. GPT-4 is the best performer, particularly when given explicit Big Five scores rather than raw text, and it handles Extraversion and Neuroticism most clearly — traits that tend to be the most linguistically marked. Open-source models (Gemma, Llama 3.1, Mistral) fluctuate more with context length and show less stable behavior. Architecturally, this is a prompt-based evaluation study, not a fine-tuning paper. The models receive zero-shot or few-shot prompts containing user posts and are asked to complete questionnaire items on a Likert scale. No training occurs. The method leans entirely on in-context learning and the models' existing personality-relevant representations. This design choice is both the study's strength (it tests off-the-shelf capability) and its limitation (it tells us nothing about whether fine-tuning or retrieval-augmented approaches could close the gap). The integrity picture is mixed. The MyPersonality dataset is well-established in personality computing research, and the BFI-2-60 is a validated instrument — both good signs. The authors compare against meaningful baselines (no-text, population tendency) and break results out by trait-score region, which is where most prior work hides its weaknesses. But the dataset is aging (Facebook posts from the early 2010s), sample sizes per trait-score bin aren't reported in the abstract, and there's no pre-registration or code availability mentioned. The evaluation is entirely self-contained — no independent replication. The milestone question is where the digital-twin aspiration meets reality. Current models predict personality-questionnaire responses with moderate accuracy for average individuals but fail at the tails. For digital twins to be useful — in therapy simulation, user modeling, or synthetic social science — they'd need to reliably reproduce individual-level questionnaire behavior within roughly 0.5 standard deviations for 80%+ of users, including extremes. We're not there, and this paper doesn't chart a clear path to get there. The experiment the authors didn't run is fine-tuning on individual user data. If in-context learning can't bridge the gap, the natural next step is parameter-efficient fine-tuning (LoRA, etc.) on a user's posts to create a genuinely personalized model. The honest read: this is probably being saved for a follow-up paper, since the current work is positioned as an evaluation study that establishes the baseline failure modes. But it's also possible they tried it and the gains were marginal — personality prediction from text is a notoriously hard ceiling to push past.