Imagine you're a plumber who arrives at a house and the homeowner says "the bathroom isn't right." No leak report, no diagram, no work order — just a vague complaint and a live system you can't afford to break. You have to ask the right questions, poke around the pipes without flooding the basement, and figure out what "right" means to this particular person. That's the gap FDE-Gym is measuring: the distance between solving a well-specified ticket and actually delivering a working solution to a real customer. The paper's committed claim: existing AI agent benchmarks (SWE-bench, WebArena, etc.) test agents on tasks where requirements are already spelled out. Real customer delivery — the kind forward-deployed engineers do — requires discovering unstated constraints from ambiguous conversations, operating on live systems, and delivering measurable business improvement. FDE-Gym is the first benchmark built around this full loop, with 95 tasks across 18 industry sectors, scripted stakeholders, and a scoring metric tied to actual business uplift. The numbers are sobering. Thirteen model configurations from seven providers were tested. The best configuration scored a mean of 0.56 on the normalized business-uplift metric and fully resolved (score ≥ 0.8 with no incidents) only 14 of 95 tasks — a 14.7% completion rate. The primary failure mode is information discovery: models struggle to ask the right questions and uncover constraints that stakeholders haven't volunteered. But here's the sharper finding — even on the 23 tasks where the best model successfully obtained every key fact from stakeholders, it still only hit the 0.8 threshold on 10 of them. Knowing the requirements is necessary but not sufficient; translating knowledge into working solutions on production-like systems is a separate, unsolved problem. The benchmark architecture is well-constructed for its purpose. Each of the 20 customer scenarios was adapted from real delivery projects by industry experts, not synthesized from academic templates. Agents interact with scripted stakeholders (sending messages, requesting trial runs) and operate on modifiable customer systems. The scoring function penalizes excessive interaction — messages, trial runs, waiting time — which prevents brute-force approaches. An online platform lets human FDEs practice under identical grading, creating a human baseline for comparison. The integrity picture is mixed. On the positive side: the benchmark is public, code is released, and the evaluation framework is transparent with clear scoring mechanics. On the negative side: the tasks were designed by the same team evaluating the models, the 95-task corpus is small enough that variance matters, and there's no independent replication. The scripted stakeholders, while a practical necessity, introduce a question about how well synthetic conversations proxy for real human ambiguity. The paper acknowledges these limitations but doesn't quantify them. The field fight here is real and consequential. One side — represented by SWE-bench, HumanEval, and the coding-benchmark ecosystem — argues that AI agents are rapidly approaching human-level software engineering on well-specified tasks. The other side argues that specifying the task IS the hard part, and benchmarks that hand agents a clear ticket are measuring the wrong thing. FDE-Gym plants its flag firmly on side B. The 0.56 mean score and 14.7% resolution rate are designed to recalibrate expectations about how close AI agents are to replacing the humans who do ambiguous, customer-facing technical work. The successor question is obvious: what happens when you scale from 95 tasks to 500+, add real (non-scripted) stakeholder interactions, and test on genuinely adversarial customer environments where the system fights back? The authors almost certainly didn't run real-stakeholder experiments because of reproducibility concerns — you can't grade models consistently if the human on the other end changes their mind between runs. But that's exactly the experiment that would tell us whether FDE-Gym's synthetic stakeholders are a valid proxy or a flattering substitute for the real thing.
