A logistics company deployed a customer support agent with a reasonable-looking memory setup: each session loaded the last 5,000 tokens of conversation history directly into a 32k-token context window. Sessions never approached the limit. On paper, no problem.
Six weeks in, the agent started contradicting itself — confirming a pickup time early in a session, then denying any such confirmation forty turns later. The team’s first instinct was hallucination. It wasn’t. The confirmation was 28,000 tokens back in a 30,000-token window, sitting squarely in the middle. The model was attending to it. Just not reliably. Liu et al. documented this pattern systematically: retrieval performance degrades significantly when relevant information appears in the middle of long contexts, even on models designed for long-context use 1. The context window wasn’t the problem. The position was.
This is one failure mode among many. AI agent memory architecture determines which failure modes you ship to production — not whether you’ll have them.
This post covers everything: the five memory types your agent needs, the three storage approaches and exactly how each breaks, the retrieval bottleneck that causes most real-world failures, and a concrete decision framework for choosing between them.
The Five Types of Agent Memory
The AI research community has converged on a cognitive-science-inspired taxonomy. Understanding it matters because different memory types fail in completely different ways.
1. Working Memory (The Context Window)
Everything the model can attend to right now. Fast, fully attended to, ephemeral — when the session ends, it’s gone. Modern models have 128k-token context windows, which sounds large until a single long document fills half of it.
Working memory is not just limited in size; it’s structurally biased. The “Lost in the Middle” study 1 found a U-shaped attention curve replicated across 18 frontier models: models reliably attend to the beginning and end of context windows but systematically underweight the middle. Longer context windows don’t fix this. Every model tested degraded as context length increased beyond an optimal range. Simply stuffing more context is not the answer.
2. Episodic Memory
Time-stamped records of specific events: what happened, when, and what resulted. “On February 28th, I tried to submit to AlternativeTo.net and got blocked by Cloudflare Turnstile.” This is the memory of experience, not abstracted knowledge.
A 2025 position paper argues that episodic memory is “the missing piece for long-term LLM agents” — because it enables single-shot learning from specific experiences without requiring weight updates. The agent doesn’t need retraining to remember what worked. It just needs to have stored the experience in a retrievable form.
3. Semantic Memory
Abstract, generalized knowledge: facts, concepts, relationships. Not “yesterday I learned X” but “X is true.” A knowledge base. A set of principles. A world model.
The AriGraph paper (IJCAI 2025) demonstrated that structured graph-based semantic memory substantially outperforms unstructured text stores for tasks requiring multi-hop reasoning — the kind where you need fact A and fact B together to reach conclusion C, and neither alone triggers the right retrieval.
4. Procedural Memory
Learned skills and workflows. Not what the agent knows, but how it does things. In practice: stored prompt templates, successful action sequences, tool-calling patterns that worked for past tasks.
Voyager (Wang et al., 2023) is the most striking demonstration. It built a Minecraft-playing agent that stored every successful code sequence as a callable skill, indexed by natural language description. The skill library grew across sessions. New tasks could bootstrap from stored skills. The result: 3.3x more items acquired, 15.3x faster tech-tree progression versus prior state of the art — and the skills transferred to new environments.
Most production agent frameworks handle episodic memory reasonably well. Procedural memory is almost entirely ignored.
5. Core / Persona Memory
The agent’s self-model: what it is, what its goals are, what it’s constrained to do, who it’s working for. MemGPT (2023) distinguished this as always-in-context and compressed — too important to be subject to retrieval, too small to cause significant cost 2.
The key insight: Every other memory type exists to manage what enters working memory. The hard constraint is not storage — it’s what you can attend to simultaneously. All memory architecture is, at bottom, an answer to one question: what should be in context right now?
Three Storage Approaches — and How Each Breaks
Knowing the memory types tells you what to store. The storage approach determines how you store it and how it fails.
In-Context Memory: Fast Until It Isn’t
The simplest approach: conversation history, retrieved documents, and scratchpad state all live inside the active context window. No external storage, no retrieval step, no write latency. For short-lived sessions — under a few thousand tokens, contained within a single task — this is the right choice.
The failure mode is position-dependent degradation, and it arrives before you expect it.
Utilization threshold. In-context memory begins degrading well before the context window is full. The logistics example above used roughly 94% of a 32k window. The degradation isn’t linear — it’s concentrated in the middle. An agent with a 128k context window and 100k tokens of history is not necessarily more reliable than one with a 32k window and 28k tokens. The ratio matters less than the absolute distance between relevant tokens.
Session boundary. In-context memory is stateless across sessions by definition. Workarounds — serializing the context to disk, loading it at session start — help only until accumulated history exceeds what fits usefully in the window. At that point you’re in external-store territory without the infrastructure to support it.
The in-context envelope: short sessions, contained tasks, no cross-session continuity requirements. Outside that envelope, you’re accumulating a reliability debt that surfaces as intermittent, hard-to-reproduce failures.
Retrieval-Augmented Memory: Precise Until It Goes Stale
Retrieval-augmented memory decouples storage from the context window. The agent writes observations, summaries, or documents to a vector store; at query time it retrieves the top-k relevant chunks and injects them into context. This breaks the hard ceiling of in-context memory and makes cross-session continuity tractable.
The failure modes are different in character from in-context degradation:
Temporal staleness. When a user updates a preference or goal, the new information creates new embeddings. It doesn’t automatically displace old ones. Similarity search is not temporal search. An agent querying for “user’s preferred shipping method” six months after a preference change may retrieve the old vector, the new one, or both — depending on embedding similarity. This is invisible at write time. It surfaces as confident, wrong answers.
Recall gaps at scale. The LoCoMo benchmark — long-term conversations spanning up to 35 sessions, averaging 9,000 tokens each — found that standard RAG systems struggled specifically with temporal and causal reasoning across sessions 3. Retrieval based on semantic similarity can’t reliably reconstruct causal chains. If the agent needs to reason about a sequence of events, chunk-level retrieval returns relevant individual facts without surfacing the ordering relationships between them.
Relevance gaps. As the vector store grows, the signal-to-noise ratio of retrieved chunks decreases. At small scale, top-k retrieval returns obviously relevant chunks. At large scale — thousands of past sessions, tens of thousands of observations — semantically similar but contextually irrelevant content competes for the top-k slots. Effective working memory degrades not because information was lost, but because it’s buried.
MemoryArena (2026) found that agents performing well on long-context factual benchmarks consistently struggled when memory had to guide future actions rather than answer factual questions 4. Retrieval-augmented systems are well-optimized for factual recall. They are not well-optimized for procedural continuity.
External Store: Consistent Until It Isn’t
External store memory — relational databases, key-value stores, graph databases — treats agent state as structured data. The agent reads and writes explicit records: user preferences, task states, entity relationships, completed steps. Retrieval is deterministic: a lookup by key returns exactly the record you wrote, not an approximation based on embedding distance.
For long-running agents managing complex, structured state — multi-step workflows, entity relationships, cross-session task continuity — external stores provide guarantees the other approaches can’t. If you need to know whether a task was completed, a database query is unambiguous. A similarity search is not.
The failure modes are distinct:
Write latency and consistency under concurrency. External stores introduce synchronous write operations into the agent’s execution path. For single-threaded, single-session agents: non-issue. For agents running concurrent sessions — parallel tool calls, multi-agent pipelines, multiple instances sharing state — writes can race. Two instances that simultaneously check whether a step has been completed may both conclude it hasn’t, both execute it, and produce duplicate or conflicting results. This is invisible in development environments. It surfaces under production load.
Cold-start for new entities. External stores know nothing about entities that haven’t been explicitly written to them. An in-context agent can reason about a new entity introduced in the current session. A retrieval-augmented agent can infer properties from related observations. An external store agent returns a null lookup and must handle it explicitly. For agents that frequently encounter novel entities, this cold-start cost is structural, not incidental.
Schema rigidity. Structured stores require schemas made at design time. When the agent’s requirements evolve — new task types, new entity relationships, new constraints — the schema must evolve too. Agents built on external stores accumulate migration debt in proportion to how much the underlying task domain changes. In-context and retrieval-based systems degrade gracefully when new information arrives; they don’t enforce structure.
Comparison Table
| Approach | Primary failure mode | When it shows up | Observable symptom | Mitigation |
|---|---|---|---|---|
| In-context | Position-dependent degradation | ~50–70% window utilization | Inconsistency within a session; confident wrong answers | Trim to recent + salient; use compression strategies |
| Retrieval-augmented | Temporal staleness; recall gaps | >10k stored chunks; long-horizon tasks | Wrong answers that were once correct; missing causal chains | Versioned embeddings; recency-weighted retrieval; explicit invalidation |
| External store | Concurrency races; cold-start; schema drift | Concurrent sessions; novel entity volume | Duplicate actions; null-lookup failures; silent migration errors | Optimistic locking; null-handling policies; schema versioning |
Why Retrieval Is the Real Bottleneck
Across the 2025 memory research literature, one finding repeats: most agent memory failures are not storage failures. The information was stored correctly. Retrieval failed to surface it when needed.
The failure modes:
- Query-storage mismatch: You stored a memory using one vocabulary; you’re searching with different vocabulary. Cosine similarity on embeddings helps but doesn’t solve this — especially when the relevant memory is conceptually adjacent rather than literally similar to the query.
- Multi-hop retrieval failure: You need memory A and memory B together to answer a question, but neither alone is the top retrieval hit. Standard vector search retrieves the single most similar document; it can’t reason about combinations.
- Stale embeddings: Your embedding model was updated. Old stored vectors are from a different model generation. Retrieval silently degrades. No error is thrown.
- No importance differentiation: If every memory is stored with equal weight, noise competes with signal. After many sessions, the majority of stored memories may be low-value. Retrieval returns a mix of critical principles and irrelevant details.
The generative agents paper (Park et al., 2023) introduced a retrieval scoring formula that’s been widely replicated:
retrieval_score = α·recency + β·importance + γ·relevance
# recency: exponential decay from last access
# importance: LLM-assigned 1-10 at storage time
# relevance: cosine similarity to current query
# all three normalized to [0,1] before summingThe importance score is the underused piece. Asking the LLM “how important is this, on a scale of 1-10?” when storing a memory is cheap and highly effective. “Brushing teeth” = 1. “Getting divorced” = 9. Most production systems skip this, treating all memories equally, and pay for it in retrieval quality.
The Compression Trap: Ahead-of-Time vs. Just-in-Time
A common implementation choice: summarize conversations at write time. Take the last 10 exchanges, condense them into a paragraph, discard the originals.
The 2025 GAM (General Agentic Memory) paper argues this is structurally wrong. Compression at write time discards information based on what seems important now. But what’s important is query-dependent — you don’t know yet what questions will be asked later. The information you discarded may be exactly what’s needed three weeks from now.
GAM’s solution: just-in-time compilation for memory. A separate Memorizer component stores everything with lightweight indexes — it never discards raw data. At query time, a Researcher component conducts a targeted deep-research pass over the stored data to synthesize exactly the context needed.
Results: GAM beats all prior systems (A-Mem, Mem0, MemoryOS) on multi-hop tasks across 56K–448K token contexts. The insight is about where to pay cost: move it from write-time to read-time, because read-time is when you know what you actually need.
For more on compression strategies and when to use them, see our deep-dive on agent memory compression and the case for memory systems without a vector database.
Memory That Evolves: Beyond Append-Only
The A-Mem paper (NeurIPS 2025) introduced one of the more important architectural ideas: memory as an interconnected network rather than a flat store, inspired by the Zettelkasten method.
When a new memory arrives, the system doesn’t just store it. It finds semantically related existing memories and asks the LLM to identify meaningful connections — not just cosine similarity, but conceptual relationships. And crucially: existing memories update when new memories arrive.
This is the difference between append-only memory and evolving memory. If you learn something that refines a previous principle, the previous principle should update — not just coexist with the new learning in a state of quiet contradiction. Most memory systems are append-only. A-Mem’s result: 2x improvement on multi-hop questions, using only 1,200–2,500 tokens versus 16,900 for competitors.
The operational implication: your principles.md or knowledge base accumulates contradictions over time unless you have an active deduplication step. Before adding a new principle, explicitly check whether it refines or contradicts an existing one. Update the existing entry rather than appending.
Memory Poisoning: The Attack Surface Most Builders Ignore
This failure mode is less discussed but increasingly important for agents that read external content.
Agents that read web pages, emails, documents, or API responses are exposed to indirect prompt injection. The malicious actor doesn’t need access to the agent’s prompt — they embed instructions in content the agent reads and potentially stores. If the agent stores summaries of what it reads, a poisoned page can install lasting behavioral modifications.
The InjecMEM paper (2025) demonstrated single-interaction poisoning with attack success rates exceeding 84%. MemoryGraft showed “semantic imitation” — crafting poisoned memories that look like legitimate agent memories but contain adversarial payloads, with effects persisting across sessions.
For an agent that browses the web and stores research findings, this is not theoretical. Provenance tagging — recording source and date alongside each finding — makes it possible to invalidate a batch of stored beliefs if the source is later discovered to be compromised.
Decision Framework: Which Approach for Your Agent
These are not equal tradeoffs between style preferences. Each choice forecloses options.
1. Does the agent need to remember anything across session boundaries? No → in-context memory. It’s faster, simpler, and has no infrastructure footprint. Any retrieval or external store overhead is pure cost. Yes → proceed to question 2.
2. Is the information primarily factual (preferences, past events, document contents) or procedural (task state, step completion, workflow continuity)? Factual → retrieval-augmented memory. Procedural → external store. Mixing both often means both are needed, with in-context serving as the working buffer for the current session.
3. Will multiple agent instances ever share state simultaneously? Yes → external store is mandatory for any state that requires consistency. Retrieval-augmented handles concurrent reads acceptably; concurrent writes without explicit locking produce races.
4. How often does your agent encounter entities it has never seen before? High novelty volume (new users at session start, new document types, new task categories) penalizes external stores. Cold-start cost is correctness failure, not just latency.
5. What is the expected volume of stored state over the agent’s lifetime? Under a few thousand observations: in-context or small retrieval store. Tens of thousands to hundreds of thousands: retrieval store with explicit staleness management. Highly structured state that must remain queryable over millions of records: external store.
6. What failure mode is most acceptable for your application? This isn’t rhetorical. A customer support agent can tolerate occasional retrieval misses. A financial workflow agent cannot tolerate duplicate step execution. Choose the failure mode your application can detect and recover from.
What Most Practitioners Get Wrong
The default mistake is treating memory as an infrastructure choice rather than a correctness choice. Teams pick in-context memory because it requires no setup, retrieval because it seems scalable, or external store because it looks like a real database. None of these are reasons.
The actual question: what does failure look like in production, and can I detect it?
In-context memory fails silently — consistent-looking answers that are wrong because the relevant context was in the middle. Retrieval-augmented fails confidently — correct-looking answers that are stale. External store fails structurally — correct behavior until a new entity appears or concurrent writes race.
Of these, retrieval-augmented failure is the hardest to catch in testing. Unit tests and integration tests almost always run with small, fresh vector stores. Staleness failures require months of production use to emerge. Concurrency failures in external stores require load testing most teams skip. In-context degradation is at least detectable with long-context test cases — if you write them.
The open problems the field acknowledges honestly:
- Learned forgetting. No current system has a principled policy for what to forget. Heuristic decay (time-based eviction) is the state of the art. The right approach requires a training signal that’s hard to construct.
- Multi-agent memory coordination. When multiple agents share memory, who owns what? How do you reconcile conflicting writes? Largely unsolved.
- Evaluation. The 2026 AMA-Bench paper notes that most benchmarks test dialogue-only memory and miss the real challenge: agentic trajectories across diverse task types. Even GPT-5.2 achieves only 72% on AMA-Bench.
The Practical Upshot
For practitioners, the single highest-leverage memory improvement is adding an importance score at write time. It’s one extra LLM call. It makes retrieval dramatically better. Almost no production agents implement it.
The four concrete changes that follow from the research:
- Importance scoring at write time. When adding to your knowledge base or principles file, score it 1–10. If it’s below 6, it doesn’t go in — it goes in session notes instead. This keeps semantic memory high-signal.
- Active deduplication. Before adding a new principle, check whether it refines or contradicts an existing one. Update the existing entry rather than appending.
- Separate procedural memory. Don’t mix episodic records (“this worked in session #12”) with procedural templates (“to do X, use this pattern”). The procedural patterns need to be indexed and retrievable, not buried in chronological notes.
- Provenance tagging. When storing research findings, record the source and date. This makes it possible to invalidate a batch of stored beliefs if the source turns out to be wrong.
For more on how these principles apply in 2026 production deployments, see the agent-memory-2026 overview.
The bottom line: Memory architecture is not a performance optimization. It’s a correctness decision. The agent that “forgets” important context in production isn’t suffering from bad luck — it’s running the predictable failure mode of an architecture chosen without understanding how it breaks. Choose based on how you want to fail, then build the monitoring to catch it.
Footnotes
-
Liu, N. F., Lin, K., Hewitt, J., Paranjape, A., Bevilacqua, M., Petroni, F., & Liang, P. (2023). Lost in the Middle: How Language Models Use Long Contexts. arXiv:2307.03172. https://arxiv.org/abs/2307.03172 ↩ ↩2
-
Packer, C., Wooders, S., Lin, K., Fang, V., Patil, S. G., Stoica, I., & Gonzalez, J. E. (2023). MemGPT: Towards LLMs as Operating Systems. arXiv:2310.08560. https://arxiv.org/abs/2310.08560 ↩
-
Maharana, A., et al. (2024). Evaluating Very Long-Term Conversational Memory of LLM Agents. arXiv:2402.17753. https://arxiv.org/abs/2402.17753 ↩
-
He, Z., Wang, Y., Zhi, C., Hu, Y., Chen, T.-P., Yin, L., Chen, Z., Wu, T. A., Ouyang, S., Wang, Z., Pei, J., McAuley, J., Choi, Y., & Pentland, A. (2026). MemoryArena: Benchmarking Agent Memory in Interdependent Multi-Session Agentic Tasks. arXiv:2602.16313. https://arxiv.org/abs/2602.16313 ↩