Imagine you have a race car with the engine welded shut. You can't tune the motor — it's sealed, inspected, and certified. But you CAN redesign the chassis, the suspension geometry, the aerodynamics, and the tire compound. CLEO is that chassis redesign for LLM agents: given a fixed, self-hostable language model that security teams have already vetted, it searches the space of agent harness configurations — which tools to expose, what guardrails to enforce, how memory works, how the reasoning loop iterates — and keeps only the components that improve accuracy on a validated dataset while remaining auditable. The core claim is straightforward but practically important: you don't need frontier-scale hosted models to do investigative analysis if you're willing to invest engineering effort in the scaffolding around a smaller model. CLEO reaches an 83.8% pass rate on a held-out evidence-analysis dataset, landing within 1.2 percentage points of the strongest Claude Code SDK baseline. The model underneath is two orders of magnitude smaller than Claude. That's the headline number, and it matters because government intelligence teams — the explicit audience — face hard constraints around hosting models on classified networks where API calls to Anthropic or OpenAI are not an option. The method belongs to the agent-harness optimization family rather than the model-training family. CLEO doesn't fine-tune or distill the underlying LLM. It treats the model as a fixed black box and searches over the combinatorial space of harness components: tool selection, guardrail policies, memory management strategies, and loop architectures. The optimization criterion is accuracy on an expert-validated dataset, but with a generality constraint — components that only help on narrow cases get pruned. This is closer to AutoML for agent scaffolding than it is to RLHF or prompt engineering. The baseline comparison is the strongest card in the paper: Claude Code SDK, presumably running a frontier Claude model, is named explicitly, and the gap is stated as 1.2 points. That's honest positioning — they're not claiming to beat the best hosted model, they're claiming near-parity at radically lower scale. The weakness is that we only see one baseline and one dataset. There's no comparison against other harness-optimization approaches (AutoGen, DSPy's module optimization, CrewAI configurations), and the dataset itself is described as expert-validated but not publicly available. Integrity is the soft spot. This is a 3-page industry talk paper, not a full research submission. The dataset is proprietary and expert-validated internally. There's no code release mentioned, no pre-registration, and no independent replication. The single held-out test set is the only evaluation surface. For a government-adjacent team presenting at an industry day, this is expected — but it means the 83.8% number carries a trust-me asterisk until someone reproduces it on a public benchmark. The practical implication is clear: organizations that cannot use hosted frontier models have a concrete methodology for closing most of the gap. The 1.2-point delta at 100× smaller scale is the kind of result that changes procurement conversations. If CLEO's approach generalizes beyond this specific dataset and task, it suggests a design pattern — harness optimization as a substitute for model scaling — that could reshape how constrained-environment deployments work. The obvious next experiment is running CLEO on public benchmarks (HotPotQA, FRAMES, or similar multi-hop reasoning tasks) with an open-weight model like Llama or Mistral, so the community can verify the claim independently. The authors almost certainly didn't do this because the dataset and task are tied to their government client's specific evidence-analysis workflow. That's legitimate — but it means the paper is more of a proof-of-concept report than a reproducible research contribution.
