Memory
An LLM call is stateless. The model retains nothing between requests, so any continuity a user experiences is something the application supplied by putting prior information back into the context.
"Memory" is therefore not a model feature. It is a storage and retrieval design decision, with the same trade-offs as any other cache.
The Four Memory Types
Short-term and long-term describe where memory lives. They say nothing about what it holds. Four types are worth naming separately, because each one is stored differently and each one fails differently.
| Type | Holds | Where it lives | Fails as |
|---|---|---|---|
| Working | The current task: turns so far, intermediate tool results, the context retrieved for this question | The context window | Truncation. Early turns drop out with no signal to anyone |
| Episodic | Specific past events: what was asked, what was decided, what the outcome was | Session summaries or a vector store, retrieved by recency or similarity | Compounding inaccuracy as summaries summarise summaries |
| Semantic | Stable facts: role, region, reporting currency, entity relationships, domain rules | A profile store or knowledge base, looked up directly | Stale values asserted confidently long after they changed |
| Procedural | How the system does things: prompt templates, tool definitions, few-shot examples, learned routines | Version-controlled prompts and tool schemas, occasionally fine-tuned weights | Silent drift when a prompt changes and nothing re-evaluates |
Working memory is what the short-term section below covers. Episodic and semantic are what the long-term section covers. Procedural is the one most teams never label as memory at all, which is exactly why prompt changes so often ship without an evaluation run behind them.
Two Different Problems
| Aspect | Short-term memory | Long-term memory |
|---|---|---|
| Scope | One task or conversation | Across sessions, indefinitely |
| Holds | Turns so far, intermediate tool results, working state | Preferences, stable facts, prior decisions, entity history |
| Lives in | The context window | A database, retrieved selectively |
| Bounded by | The context window and cost per call | Storage, and retrieval precision |
| Fails as | Truncation and forgotten early turns | Stale facts asserted confidently |
Conflating them is the usual mistake. Appending everything to the conversation history is not long-term memory, it is a context window overflow with extra steps.
Short-Term Patterns
| Pattern | When to use | Cost |
|---|---|---|
| Full buffer | Short, bounded interactions under about ten turns | Grows linearly, and every prior turn is re-billed on every call |
| Sliding window | Long conversations where only recent context matters | Early turns disappear with no signal to the user |
| Rolling summary | Long conversations where early decisions still matter | One extra model call per compaction, plus drift as summaries summarise summaries |
The default choice for a support or analytics assistant is a sliding window with a rolling summary behind it: recent turns verbatim, everything older compressed once.
Long-Term Patterns
| Pattern | Stores | Retrieval | Watch for |
|---|---|---|---|
| Profile store | Explicit structured facts: role, region, reporting currency, preferences | Direct lookup by user or entity id | Stale values. Store a timestamp and a source with every fact. |
| Vector store | Embeddings of past exchanges or extracted statements | Semantic similarity against the current question. See Embeddings | Retrieving something superficially similar but no longer true |
| Session summaries | One generated record per conversation | Recent-first, or by similarity | Compounding inaccuracy, since a summary of a summary loses the qualifiers first |
A profile store is the one worth building first. It is cheap, explicit, auditable, and covers most of what users actually mean when they say the assistant should remember them.
Writing to Memory
The hard part is not storage, it is deciding what deserves to be stored.
- Write stable facts, not passing statements. "Reports in EUR" belongs in memory. "Looking at Q3 right now" does not.
- Prefer explicit capture over inference. A fact extracted by a model from conversation is a guess. Confirmed preferences and system-of-record values are not.
- Timestamp and attribute everything. A retrieved fact with no date cannot be judged stale, and the model will assert it with full confidence either way.
- Make it correctable and deletable. Users need to see what the system believes about them and fix it, and deletion is frequently a legal requirement rather than a feature.
- Never store secrets or credentials. Anything in memory can be retrieved into a prompt, and anything in a prompt can end up in an output.
Memory also widens the trust boundary. A stored fact is untrusted input the next time it is retrieved, in exactly the way a tool result is, so it should not be treated as an instruction just because the system wrote it.
Cost and Accuracy
Memory is paid for on every call it touches, at the input token rate. Retrieving twelve past exchanges to answer a question that needed one is the same mistake as over-retrieving in RAG, and it has the same fix: retrieve broadly, rank, then send few.
Accuracy degrades in a specific way that is worth naming. As stored facts accumulate, the odds rise that a retrieved fact is out of date, and there is no mechanism by which the model detects this. Bound the store, expire entries, and re-confirm anything consequential rather than trusting it indefinitely.
Key Takeaways
- The model is stateless. Memory is application-side storage and retrieval, not a model capability.
- Four types, not two: working, episodic, semantic, and procedural. Procedural memory is the one teams forget they have.
- Keep short-term and long-term memory separate. Appending everything to history is an overflow, not memory.
- Sliding window plus rolling summary handles most conversational cases. A profile store handles most cross-session ones.
- Store stable, timestamped, attributed facts. Prefer confirmed values over model-inferred ones.
- Make memory visible, correctable, and deletable, and keep secrets out of it entirely.
- Retrieved memory is untrusted input and can be stale. Re-confirm anything consequential.