Use Your Simulator as a Spell-Check, Not a Ghost-Writer: Control Variates for Sim-to-Real RL

Instead of mixing sim data into training (and importing sim bias), this paper uses simulators purely to reduce variance — a classical statistics trick applied to actor-critic RL. NeurIPS 2026 poster.

Imagine you're estimating the average commute time in your city by surveying real commuters. It's expensive — you can only survey 50 people. Your friend offers you a traffic-simulation app that can generate 10,000 synthetic commutes for free. The naive move is to just average everything together — but if the simulation systematically underestimates rush-hour congestion, you'll get a biased answer that's worse than your 50 real samples alone. The smarter move: use the simulation only to understand the *shape* of variance in commute times, then subtract that known pattern from your real data to get a tighter estimate without importing the sim's bias. That's a control variate, and it's exactly what this paper does for reinforcement learning. The committed claim: simulation data should never be allowed to bias the learning target in RL. Instead, simulated rollouts should serve exclusively as control variates — statistical instruments that reduce the variance of policy-evaluation estimates computed from real data. The authors formalize this for temporal-difference (TD) learning in policy evaluation, then extend it to actor-critic policy optimization and offline RL. The key theoretical guarantee is that their estimator remains unbiased with respect to the real environment regardless of how bad the simulator is, while strictly reducing variance whenever the simulator has any correlation with reality. The architecture sits squarely in the actor-critic family with a TD-learning backbone. The control-variate mechanism works by computing a TD error in the simulator, measuring its correlation with the real TD error, and subtracting the correlated component. This requires paired or matched trajectories (same state-action pairs evaluated in both environments), which is a practical constraint the paper acknowledges but doesn't fully stress-test. The compute overhead is modest — one additional forward pass through the simulator per real transition — and the method is agnostic to the simulator's fidelity. The ladder comparison is against direct simulation-mixing baselines (SimQ, H-step returns with mixed data) across multiple MuJoCo tasks including HalfCheetah, Hopper, Walker2d, and Ant. The paper shows that standard mixing degrades performance when simulator bias is non-trivial, while the control-variate approach matches or beats real-data-only baselines with fewer real samples. The honest caveat: gains are most pronounced when the simulator is moderately accurate — if the sim is either nearly perfect (just use the sim) or completely uncorrelated (the control variate reduces to zero benefit), the method's advantage narrows. They don't compare against the most aggressive sim-to-real transfer methods (domain randomization, system identification) which attack the problem from a different angle. Integrity is decent for a methods paper. MuJoCo is a standard community benchmark, the experimental protocol covers multiple environments with varied simulator fidelity levels, and code is released. The main weakness is that all experiments are simulation-to-simulation: they create an artificial "real" environment by perturbing a MuJoCo sim, then use the unperturbed sim as the simulator. No physical robot experiments. The validation is therefore somewhat circular — you're testing whether a sim can help with a slightly-different sim, not whether a sim can help with actual hardware. The milestone question is about sample efficiency in real-world robotics. Today, most sim-to-real RL still requires domain randomization or fine-tuning. If control variates can reliably cut real-world sample requirements by 2-5× across diverse manipulation and locomotion tasks on physical hardware, that's the threshold where this technique enters standard practice. The paper demonstrates the mechanism; the physical validation is the gap. The obvious missing experiment is a real robot. The authors almost certainly know this — the co-authors span CS and robotics (cs.RO is a listed category), and the paper's framing screams "we want to do this on hardware." My honest read: they're saving the hardware experiments for a follow-up, or the engineering overhead of paired real-sim rollouts on physical systems proved too expensive for this submission cycle. Either way, the MuJoCo-only validation is the paper's biggest limitation and they know it.