Naive recall
Was the fact stored?
A direct question explicitly cues the personal fact and tests whether it survived the memory pipeline.
A benchmark for long-term agent memory
Keep It InMind.
A memory can be essential to a query without resembling it. InMind exposes the implicit-association blind spot—when world knowledge makes a stored fact matter, but query-time retrieval never surfaces it.
125 tasks · 10 life domains · English evaluation set
Stored memory“I have a tree-nut allergy.”
hidden bridgemacarons → almond flour
Later query“Any good macaron recipes?”
MISSED BY RETRIEVAL
125audited tasks
10life domains
38sessions after injection
84→16application gap (%)
The blind spot
Remembered on demand.
Forgotten when it matters.
Retrieval works when the query resembles the memory it needs. InMind targets the harder case: relevance exists only after a model supplies a piece of world knowledge.
1Query arrivesMacaron recipe?
2Retriever selectsBefore reasoning
3Model reasonsOver selected memory
The component that could recognize the bridge—the language model—sees the memory only after retrieval has already judged it irrelevant.
The benchmark
One task.
Four clean diagnoses.
A wrong indirect answer could mean the fact was never stored, the model lacks the bridge, retrieval missed it, or the answer ignored it. InMind separates these explanations instead of collapsing them into one score.
In-context control
Can the model bridge it?
The target memory is placed in context, isolating the model's world knowledge and reasoning ability.
Target recall
Did retrieval surface it?
An answer-blind judge checks whether the decisive fact reached the context used for the indirect query.
Application
Did the answer use it?
The final response is judged for whether it applies the source-grounded bridge appropriately.
Main finding
The facts are stored.
They simply never arrive.
Six vector, graph, and agentic memory systems can recall the same facts on demand at up to 100%, yet no query-time configuration applies them above 16%.
Why retrieval struggles
Similarity offers
no reliable signal.
Under an independently selected embedding model, InMind's target memories overlap with the background pool. The decisive memory is not consistently closer to the query than irrelevant history.
The open problem
Memory needs routing,
not just retrieval.
Retrieval preserves a lossless record at scale. Always-visible state lets a model reason jointly over distant facts and the current query. Practical systems need both—and a principled router that decides what must remain visible before the future query is known.
Long-term recordEverything retained
ROUTER
Persistent stateDecision-critical facts
Query-time searchFine-grained history
Open benchmark
Built for diagnosis,
released for research.
The current release contains the benchmark definition, canonical English dataset, JSON Schema, source traces where available, and integrity checks. Evaluation code and paper-aligned outputs will follow.
Domain distribution125 tasks
Health & wellness
46
Professional & career
26
Relationships
16
Financial
8
Legal
7
Spirituality
7
Consumer
5
Parenting
4
Personal development
3
Other
3
125 synthetic tasks
Canonical English JSONL, stable IDs, ten domains, and paired direct/indirect queries.
Browse records ↗Auditable by design
113 source-grounded tasks and 12 expert-authored tasks, with provenance retained where available.
Read the protocol ↗Paper
Keep It InMind: Benchmarking the Implicit-Association Blind Spot in Agent Memory
The full paper is now available on arXiv. Read the methods, experiments, limitations, and appendices in the paper PDF ↗