You know how airlines don't let new pilots fly a 737 full of passengers on their first day? They put them in a simulator first — not because the simulator is real, but because the mistakes are cheap. This paper does the same thing for customer-service AI agents at Nubank, Brazil's largest digital bank with 140 million customers. The core mechanism: build synthetic customers that behave like real ones, let your candidate AI agent serve them in a sandbox with simulated tool backends, score the conversations, and only promote the winners to production. The committed claim is straightforward but consequential: simulation-guided screening of CX agent configurations produces version-level quality scores that correlate with real production outcomes, and acting on those scores delivers measurable gains in live A/B tests. This is not a theoretical framework paper — it reports four deployed versions of Nubank's Card Delivery and Card Management agents, the latter being the bank's highest-volume chat support channel in Brazil. The results are concrete. Simulated binary evaluator scores showed high correlation with production scores across the four deployed versions. When the team used simulation to iterate on agent design, the winning configuration increased transactional Net Promoter Score (tNPS) by 36.69 points in a live A/B test. In a separate experiment screening open-weight model configurations across 16,000+ simulated conversations, the selected model boosted self-service rate (SSR) by 8.82 percentage points — the highest SSR ever observed at Nubank — with no statistically significant degradation in tNPS. Architecturally, this sits in the LLM-as-judge plus synthetic-data family. The Snowglobe simulator generates synthetic customer personas that react turn-by-turn to agent responses. Simulated tool outputs stand in for production backends, enabling multi-step agentic workflows (checking delivery status, blocking cards, etc.) without touching real systems. LLM-based evaluators score the resulting conversations on binary pass/fail metrics aligned with operational policies. The key compute property this leans on is that simulation is embarrassingly parallel — you can run 16,000 conversations overnight in a way you could never run 16,000 live A/B variants. The integrity picture is mixed in useful ways. On the strong side: the validation is production A/B tests with real customers, not just simulation-vs-simulation circularity. The correlation between simulated and production scores is demonstrated across four actually-deployed versions, not hypothetical configurations. On the weaker side: we don't see confidence intervals on the correlation claim, the sim-to-production mapping is version-level (not conversation-level), and the evaluator that scores both simulated and production conversations is the same LLM-based system — introducing shared-method bias. The paper is honest about limitations (synthetic customers can't fully replicate real user diversity), but the absence of independent replication and the proprietary nature of the system limit external verification. The milestone question is practical: at what scale of agent complexity and conversation depth does simulation fidelity break down? The paper demonstrates it works for card delivery and card management — structured, policy-heavy domains with clear success criteria. The next frontier is open-ended advisory conversations (investment advice, dispute resolution) where success is harder to define and simulate. The paper doesn't quantify where the sim-production correlation degrades, which is the number to track. The obvious experiment not run: testing simulation fidelity on agent configurations that performed POORLY in production, not just the winners. If you only validate that your simulator correctly ranked the versions you shipped, you have survivorship bias. Did simulation also correctly predict which candidates would fail? The honest read is (a) — running intentionally bad configs in production to validate the simulator's downside predictions would be irresponsible at 140M-user scale, so this gap is structural, not evasive.
