Read the paper
Jetriever: Shifting Agent Memory Retrieval from Similarity to Utility
on GitHub
EXCHANGE FIRST
+84.8%
MINILM CONTROLS REACH
+1.6%
EACH FIND
+90%
WHILE LAYA FINDS
+30%
Imagine you're a librarian, and someone asks: "What should I cook tonight?" The useful answer isn't the book most similar to that question — it's the one where this patron once mentioned they're vegetarian, allergic to nuts, and training for a marathon. Similarity search would retrieve cookbooks about cooking. Utility search would retrieve the conversation where the patron revealed their constraints. That's the core mechanism shift Jetriever proposes: stop ranking memories by how much they resemble the query and start ranking them by how much they'd actually help answer it. The committed claim: an LLM judge (here called Jev) can score every memory in the pool for decision-usefulness, and this full-pool utility ranking dramatically outperforms both sparse (BM25) and dense (MiniLM) similarity retrieval on indirect, real-world conversational queries. On InMind's 125 indirect-query tasks — where the gold evidence doesn't lexically or semantically resemble the question — Jetriever places the correct exchange at rank 1 in 84.8% of cases. BM25 manages 1.6%. MiniLM hits 0.0%. These aren't close contests; they're different categories of capability. The architecture is straightforward and arguably its greatest strength: no fine-tuning, no custom embeddings, no retrieval-augmented generation pipeline. Jev is a prompted LLM (likely GPT-4o-class based on the cost figures) that receives each stored memory alongside the current question and returns a decision score. The system scores the entire memory pool — no pre-filtering, no approximate nearest neighbor shortcuts. This is brute-force in the best sense: it trades compute for accuracy by letting the world knowledge baked into the LLM do the relevance judgment that embeddings fundamentally cannot do for indirect queries. The paper's 20-question diagnostic study is where the architecture argument gets sharp. Both Jev and a smaller open model (Qwen3.5 9B) find 90% of gold exchanges within the top 50 ranks, while a weaker model (Laya) finds only 30%. But at rank 1, Jev hits 85% and Qwen3.5 9B drops to 40%. This cleanly separates finding from ranking — most capable LLMs can identify relevant memories somewhere in their rankings, but precision at the top (the only position that matters for downstream agents) requires both world knowledge and calibrated scoring. On MemoryAgentBench, Jetriever scores 55.4% on a four-dimension metric against 44.3% for ReFind on the same 60-question sample. The integrity picture is mixed. The InMind benchmark is a real community dataset, and the authors test against reasonable baselines (BM25, MiniLM, and several LLM-based retrievers). But the evaluation is entirely self-scored — no independent replication, no pre-registration, and the MemoryAgentBench comparison uses a 60-question sample rather than the full benchmark. The cost figures ($0.00834 per InMind question, $0.03103 per MAB question) are admirably transparent but only viable because InMind memory pools are small. Full-pool scoring at scale — thousands of memories per user — would hit a linear cost wall that the paper doesn't address. The scaling question is the obvious successor experiment. What happens when memory pools grow from dozens to thousands of entries? The brute-force approach is elegant at current scale but its O(n) LLM-call cost per query makes it fragile as pools grow. A two-stage architecture — cheap pre-filter plus expensive Jev scoring on a shortlist — is the natural next step, and the authors almost certainly know this. My read: they're saving it for the next paper, because the current result cleanly demonstrates the utility-over-similarity thesis without muddying it with engineering optimizations. The deeper contribution here isn't the system — it's the framing. By cleanly demonstrating that indirect queries break similarity search and that LLM judgment fixes it, Jetriever gives the agent-memory community a new evaluation axis. The question is no longer 'how good is your embedding?' but 'how good is your judgment about what's useful?' That reframing, if it holds up at scale, matters more than the specific numbers.
The analogy
You're a librarian. Someone asks 'What should I cook tonight?' Similarity search pulls cookbooks about cooking. Utility search pulls the conversation where this patron mentioned being vegetarian, nut-allergic, and marathon-training. Jetriever is the librarian who remembers the patron, not the one who matches keywords.
Smart librarian
Recalls patron preferences and constraints to recommend the useful book, not the topically similar one
Jev scorer
LLM judges each stored memory for decision-usefulness to the current question, not semantic similarity
Honest assessment
Jetriever cleanly demonstrates that similarity-based retrieval is categorically broken for indirect conversational queries and that LLM utility scoring fixes it. The numbers are dramatic on the narrow task tested, but the evaluation is small-scale (125 + 60 questions, small memory pools), and the O(n) cost model is the unaddressed elephant. This matters most as a conceptual reframing — 'retrieve by utility not similarity' — that the agent-memory community should take seriously, even if the specific system won't survive contact with production-scale memory pools without architectural changes.
What Is CLAIMS?
63
/ 100
BedrockNews applies a structured analytical framework called CLAIMS to academic papers and technical research releases.
How CLAIMS worksClaim
3/5
Ladder
3/5
Architecture
3/5
Integrity
3/5
Milestone
3/5
Successor
4/5
What is this paper actually committing to?
Real reframing, not a first. The utility-over-similarity argument for agent memory is valuable and cleanly demonstrated, but LLM-as-judge retrieval has precedents. The contribution is the clean empirical separation.
The field fight
Should agent memory retrieval rely on embedding similarity or LLM-based reasoning?
Side A
Embedding similarity is fast, scalable, and good enough — optimize the embedding, not the retrieval
Held by: MiniLM, ColBERT, vector DB ecosystem, RAG pipelines
Side B
Similarity structurally fails on indirect queries — utility scoring via LLM judgment is necessary for real agent memory
Held by: Jetriever, ReFind (partially), LLM-as-judge retrieval work
Milestone tracker
Unlocks: Utility-ranked retrieval viable for production agents with long interaction histories
Current
125
queries / small pools
this paper
Next
1000+
memories per user
scalable utility retrieval
~1-2 papers; requires hybrid pre-filter architecture
Verified Facts (7)
On InMind's 125 indirect-query tasks, it places the gold exchange first in 84.8% of cases
Matched BM25 and MiniLM controls reach 1.6% and 0.0% when indexing user turns
Jev and Qwen3.5 9B each find 90% of gold exchanges by rank 50, while Laya finds 30%
At rank one, Jev reaches 85% and Qwen3.5 9B 40%
On a dimension-proportional random sample of 60 MemoryAgentBench questions, Jetriever's four-dimension score is 55.4%, against 44.3% for ReFind
Jev retrieval costs USD 0.00834 per InMind question and USD 0.03103 per MAB question on average
13 pages, 1 figure, 8 tables
Practical Implications (4)
If you're building conversational agents with memory, test your retrieval on indirect queries specifically — similarity metrics will show green while actual utility is broken
Full-pool LLM scoring is viable at ~$0.01/query for small memory pools (dozens of entries) but will need a pre-filter stage for production users with long histories
The finding-vs-ranking separation (90% found by rank 50, but only 40–85% at rank 1) suggests smaller models can pre-filter but you need a strong model for final ranking
Jev's decision-score prompting pattern is immediately reusable — no fine-tuning or custom infrastructure required
More latest news
Read all