The problem: More context doesn’t mean better memory
An assistant that knows us over time should remember more than what we said a few messages ago. It should carry forward our preferences, recognise when plans change and recover an important detail without asking us to explain everything again.
That ambition exposes a gap between how much information a model can accept and how reliably it can use it.
In its 2025 report on context rot, Chroma evaluated 18 language models and found that performance became increasingly unreliable as input grew, even on deliberately simple tasks. The effect varied across models and conditions, but longer input could itself make a task harder.[1]
For a personal assistant, preferences and earlier decisions become buried under a growing conversation. The model may lose track of them even while they remain inside its context window.
A bigger context window is not the answer
The Lost in the Middle study found a related problem: models often used information more effectively when it appeared near the beginning or end of a prompt than in the middle. The evidence was present, but its position affected how well the model used it.[2]
We saw this in our own experiments. With the same retrieval method, a 500,000-token working window scored 38.4 on MRCR, compared with 24.9 for an 850,000-token window. In that comparison, the larger window retained more similar replies, making the intended one harder to distinguish.
Gaps in current memory systems
Memory systems can preserve original messages, extract facts, summarise older exchanges and retrieve relevant passages. Each serves a purpose.
Summaries preserve the broad story. Extracted facts make personal details easier to find. Original messages retain wording and context that may matter later.
Retrieval methods are also improving. Anthropic’s Contextual Retrieval adds explanatory context to passages before indexing them. Google Research’s work on sufficient context examines whether retrieved material contains enough evidence to answer a question.[4][5]
For a continuing conversation, these capabilities need to work together. If a hotel booking changes, an assistant should preserve the replacement reference, understand when it applies and recover the relevant details when asked about the trip.
I switched to Harbor Hotel.
How systems keep longer sessions going
Compaction allows a conversation to continue by replacing older messages with a shorter summary. But a detail that appears unimportant during summarisation may become essential later.
Keeping everything active creates another problem. Old instructions, repeated discussions and similar alternatives accumulate alongside the information needed now. Anthropic’s work on context engineering recommends actively managing this growing context as an agent works.[3]
A continuing session needs both a manageable working context and a way to recover details that have left it.
Our vision: memory that stays useful over time
We want to build AI that develops an understanding of someone through continued interaction. It should remember preferences, follow changing plans and bring back earlier details when they become relevant.
Human memory offers a useful analogy: a limited working memory draws on a much larger store of experience.[6]
That gives our research three practical goals: preserve important details, recover the right evidence and keep the working context manageable as the conversation grows.
We began exploring these goals inside the model itself, through its KV cache.
What we learned from working with KV caches
We began by asking whether a model could reuse what it had already processed. A KV cache holds internal representations of earlier tokens, helping the model continue without processing all the same text again. Saving those representations seemed like a natural starting point for persistent memory.
Internal KV memory
Our work showed that preserving a model’s internal state and building a reusable memory are different challenges. A cache belongs to a particular model and the context in which it was created. As a conversation changes, restoring an earlier cache does not necessarily give the model the same understanding it would gain from reading that information again.
Storage and portability added further complications. KV caches can be useful for continuing computation, but using them as lasting memory requires careful handling of how and where they are restored.
The central lesson was that memory needs to remain useful after the situation that created it has changed.
Original-text memory
Original text gave us a simpler foundation. It preserves the wording of a conversation, can be inspected directly and can be read by different models. Older information can stay in an archive until a new question makes it relevant again.
This shifted our focus from preserving internal representations to managing evidence: what should stay in the model’s working context, what can move into the background, and what needs to return.
That became the basis of our adaptive API approach—keep the original information available, and adapt the active context to the task at hand.
Adaptive API: Managing memory through the API
Adaptive API sits between the application and the model. Before each call, it manages the conversation and assembles the context the model receives.
It has three parts: an archive of original messages, a search index and a bounded working window.
The archive and index form the external memory we call m. Older turns can leave the working window while their original text remains available for retrieval.
We use BM25 to search the archive. It ranks passages using matching terms, their distinctiveness and passage length. This search requires neither an embedding model nor an additional LLM call.[7]
The model’s weights and architecture stay unchanged. Adaptive API manages the text supplied through its ordinary API.
How context adapts to the question
Save the conversation. Incoming messages enter the archive. When the active window reaches its limit, older turns leave in batches while remaining searchable.
Check for earlier evidence. For each question, the system compares the strongest archived match with the strongest match still active. This helps it decide whether older material should return.
Select and restore passages. Matches are ranked, and the relative drop between neighbouring relevance scores determines how many to retain. We call this the “score knee.” Selected passages can include nearby text and return in their original conversation order.
Refill when needed. If the model requests more context, the system expands retrieval and makes another answer attempt.
A question about Friday’s hotel booking might bring back the replacement reference and arrival instructions. A question about dinner would call for a different selection.
“Repeat the third humorous poem about penguins.”
Exact text, kept in conversation order
Every message is saved as exact text. When the active window fills, older turns leave the prompt but remain in the searchable archive.
What we tested
We evaluated the approach on two benchmarks: MRCR and World-solve.
MRCR tests whether a model can recover one particular earlier reply from many similar exchanges. A question might ask for the third poem about a given subject, requiring both exact recall and identification of the correct instance.[8]
We replayed one continuous conversation spanning approximately 6.8 million tokens. Haiku and Qwen answered the same 93 questions under three setups: native compaction, native compaction with retrieval and Adaptive API.
World-solve is our benchmark for continuity across connected tasks. Each world contains linked records, changing information and misleading alternatives. Questions build on earlier answers, so one mistake can affect subsequent steps.
We evaluated 144 questions across 12 worlds, spanning approximately 2.3 million tokens. Retrieval was disabled to isolate the effect of window size.
Results
Recall across a continuing session
On Haiku, Adaptive API increased the mean MRCR score from 10.2 to 47.6, a 367% relative improvement, while reducing cost by 41% compared with native compaction.
Correct answers increased from 7 to 37 out of 93, and the run cost fell from $16.72 to $9.85. The improvement in correct answers was statistically significant within this session.
Qwen showed a similar pattern. Its mean score rose from 15.4 to 68.8, while correct answers increased from 14 to 62 out of 93.
After the first compaction, native Haiku answered only 2 of the remaining 81 questions correctly; native Qwen answered 1 of 77. Summaries allowed the conversations to continue, but often omitted the exact replies that later questions required. Restoring original passages made that evidence available again.
Compare the same 93 questions.
Mean recall score higher is better
Correct answers out of the same 93 questions
Mean scores measure response similarity on a 0–100 scale. Correct answers meet the 0.9 score threshold.
The native + retrieval result uses its best reported setting without refill; adaptive API includes refill. The paired correctness difference was not statistically significant (p = 1.0).


MRCR scores measure response similarity, not accuracy percentages. Correct-answer counts use a separate score threshold. The 367% figure describes the relative increase in mean score.
The cost of keeping context active
The stronger comparison was against native compaction with retrieval.
Once both systems could recover original text, correct-answer counts were similar: 36 versus 37 on Haiku, and 58 versus 62 on Qwen. These differences were not statistically significant.
The clearer advantage was efficiency. Adaptive API reduced Haiku’s cost from $18.20 to $9.85, approximately 46%, while peak context fell from 841,000 to 448,000 tokens.
On Qwen, average tokens retained in KV fell from 605,000 to 305,000. GPU cost and total runtime also fell by approximately half.
The comparisons differ in some refill and harness settings, detailed in the paper. Within these runs, much of the conversation could remain outside the active window while retrieval maintained comparable recall.
The cost of keeping context active.
API cost lower is better
Peak context lower is better
| Measure | Native + retrieval | Adaptive API |
|---|---|---|
| Median time per call | 6.4 s | 4.0 s |
37 correct answers versus 36, at $8.35 less for the run.
The native + retrieval result uses its best reported setting without refill; adaptive API includes refill. The paired correctness difference was not statistically significant (p = 1.0).
Finding the working-window limit
World-solve showed both the benefit and the limit of trimming context.
In the main Haiku sweep, a half-sized window retained every required record. With low thinking enabled, accuracy increased from 84.7% to 94.4%, while cost fell from $6.56 to $4.13.
With medium thinking enabled, the accuracy differences narrowed. Across Haiku, Opus and SWE-2, the half-window results stayed within four correct answers out of 144 of native, while reducing cost by 29–37%.
More aggressive trimming began removing evidence needed for the task. At one fifth of the window, Haiku’s accuracy fell to 79.2%, below native.
Because retrieval was disabled, removed evidence could not return. A smaller working window can help, but it must still reach far enough back for the task being performed.

What comes next
Retrieving evidence does not guarantee that a model will use it correctly. In 54 of Haiku’s 56 remaining MRCR misses, the requested text was already in the prompt. The model still struggled to identify or reproduce the intended reply among similar alternatives.
Our next work focuses on selecting passages more precisely and making their source, order and relationship to the question clearer. We also need more conversations, repeated runs, paraphrased questions and tasks requiring several connected facts.
The current evidence comes from one continuous MRCR session and a World-solve study that tested window management without retrieval. Within those settings, Adaptive API improved recall over native compaction and achieved similar recall to native + retrieval with substantially lower resource use.
For an assistant that develops an understanding of someone over time, this is a useful foundation: preserve the conversation, keep the working context focused and bring earlier details back when they matter.
References
- Hong, K., Troynikov, A., & Huber, J. (2025).Context Rot: How Increasing Input Tokens Impacts LLM Performance. Chroma technical report.
- Liu, N. F., Lin, K., Hewitt, J., Paranjape, A., Bevilacqua, M., Petroni, F., & Liang, P. (2024).Lost in the Middle: How Language Models Use Long Contexts. Transactions of the Association for Computational Linguistics, 12, 157–173.
- Anthropic. (2025).Effective Context Engineering for AI Agents.
- Anthropic. (2024).Introducing Contextual Retrieval.
- Rashtchian, C., & Juan, D.-C. (2025).Deeper Insights into Retrieval Augmented Generation: The Role of Sufficient Context. Google Research.
- Cowan, N. (2001).The Magical Number 4 in Short-Term Memory: A Reconsideration of Mental Storage Capacity. Behavioral and Brain Sciences, 24(1), 87–114.
- Robertson, S., & Zaragoza, H. (2009).The Probabilistic Relevance Framework: BM25 and Beyond. Foundations and Trends in Information Retrieval.
- Vodrahalli, K., et al. (2024).Michelangelo: Long Context Evaluations Beyond Haystacks via Latent Structure Queries. arXiv:2409.12640.