Memory

An LLM call is stateless. The model retains nothing between requests, so any continuity a user experiences is something the application supplied by putting prior information back into the context.

"Memory" is therefore not a model feature. It is a storage and retrieval design decision, with the same trade-offs as any other cache.


The Four Memory Types

Short-term and long-term describe where memory lives. They say nothing about what it holds. Four types are worth naming separately, because each one is stored differently and each one fails differently.

Working Memory
This call only
Working
Turns so far, tool results, retrieved context
Durable Memory
Across sessions
Episodic
What happened: past interactions, decisions, outcomes
Semantic
What is true: stable facts about users, entities, and domain
Procedural
How to do it: prompt templates, tool definitions, routines
TypeHoldsWhere it livesFails as
WorkingThe current task: turns so far, intermediate tool results, the context retrieved for this questionThe context windowTruncation. Early turns drop out with no signal to anyone
EpisodicSpecific past events: what was asked, what was decided, what the outcome wasSession summaries or a vector store, retrieved by recency or similarityCompounding inaccuracy as summaries summarise summaries
SemanticStable facts: role, region, reporting currency, entity relationships, domain rulesA profile store or knowledge base, looked up directlyStale values asserted confidently long after they changed
ProceduralHow the system does things: prompt templates, tool definitions, few-shot examples, learned routinesVersion-controlled prompts and tool schemas, occasionally fine-tuned weightsSilent drift when a prompt changes and nothing re-evaluates

Working memory is what the short-term section below covers. Episodic and semantic are what the long-term section covers. Procedural is the one most teams never label as memory at all, which is exactly why prompt changes so often ship without an evaluation run behind them.


Short-Term Versus Long-Term: When to Use What

AspectShort-term memoryLong-term memory
ScopeOne task or conversationAcross sessions, indefinitely
HoldsTurns so far, intermediate tool results, working statePreferences, stable facts, prior decisions, entity history
Lives inThe context windowA database, retrieved selectively
Bounded byThe context window and cost per callStorage, and retrieval precision
Fails asTruncation and forgotten early turnsStale facts asserted confidently
When to useAlways. Any multi-turn interaction needs it, and the only question is which pattern belowOnly when a later session genuinely needs what an earlier one learned. If nobody would notice it missing, do not build it

Conflating them is the usual mistake. Appending everything to the conversation history is not long-term memory, it is a context window overflow with extra steps.


Short-Term Patterns

Short-Term
Per conversation
Full Buffer
Every turn, verbatim
Sliding Window
Last N turns only
Rolling Summary
Old turns compressed
This Call
What reaches the model
Assembled Context
Built fresh for every call
Long-Term
Across sessions
Profile Store
Explicit key/value facts
Vector Store
Semantic recall of past exchanges
Session Summaries
One record per conversation
PatternWhen to useCost
Full bufferShort, bounded interactions under about ten turnsGrows linearly, and every prior turn is re-billed on every call
Sliding windowLong conversations where only recent context mattersEarly turns disappear with no signal to the user
Rolling summaryLong conversations where early decisions still matterOne extra model call per compaction, plus drift as summaries summarise summaries

The default choice for a support or analytics assistant is a sliding window with a rolling summary behind it: recent turns verbatim, everything older compressed once.


Long-Term Patterns

PatternStoresRetrievalWatch for
Profile storeExplicit structured facts: role, region, reporting currency, preferencesDirect lookup by user or entity idStale values. Store a timestamp and a source with every fact.
Vector storeEmbeddings of past exchanges or extracted statementsSemantic similarity against the current question. See EmbeddingsRetrieving something superficially similar but no longer true
Session summariesOne generated record per conversationRecent-first, or by similarityCompounding inaccuracy, since a summary of a summary loses the qualifiers first

A profile store is the one worth building first. It is cheap, explicit, auditable, and covers most of what users actually mean when they say the assistant should remember them.


Writing to Memory

The hard part is not storage, it is deciding what deserves to be stored.

  • Write stable facts, not passing statements. "Reports in EUR" belongs in memory. "Looking at Q3 right now" does not.
  • Prefer explicit capture over inference. A fact extracted by a model from conversation is a guess. Confirmed preferences and system-of-record values are not.
  • Timestamp and attribute everything. A retrieved fact with no date cannot be judged stale, and the model will assert it with full confidence either way.
  • Make it correctable and deletable. Users need to see what the system believes about them and fix it, and deletion is frequently a legal requirement rather than a feature.
  • Never store secrets or credentials. Anything in memory can be retrieved into a prompt, and anything in a prompt can end up in an output.

Memory also widens the trust boundary. A stored fact is untrusted input the next time it is retrieved, in exactly the way a tool result is, so it should not be treated as an instruction just because the system wrote it.


Cost and Accuracy

Memory is paid for on every call it touches, at the input token rate. Retrieving twelve past exchanges to answer a question that needed one is the same mistake as over-retrieving in RAG, and it has the same fix: retrieve broadly, rank, then send few.

The cost failure. An assistant retrieves the twelve most recent exchanges on every turn as a blunt approximation of relevance. At roughly 400 tokens each that is 4,800 input tokens per call, before the system prompt or the question. Across 5,000 calls a day it is 24 million input tokens paid daily to supply context that, for most turns, one exchange would have covered. The retrieval was never wrong, it was just never ranked. Run the arithmetic against the input-token rate for your platform before assuming memory is free: Microsoft Foundry, AWS Bedrock, Google Vertex AI, Databricks, Snowflake Cortex. See Tokens for how the count is built up.

The accuracy failure. A profile store records "reports in USD" during onboarding. Fourteen months later the client has moved to EUR reporting, nobody updated the store, and every generated commentary since has quietly labelled EUR figures as dollars. The model cannot detect this: a retrieved fact arrives with exactly the same authority whether it is current or two years stale.

Accuracy degrades in that specific way, and it compounds. As stored facts accumulate, the odds rise that a retrieved fact is out of date, and nothing in the system flags it. Bound the store, put a timestamp and a source on every fact, expire entries on a schedule, and re-confirm anything consequential rather than trusting it indefinitely.


Key Takeaways

  • The model is stateless. Memory is application-side storage and retrieval, not a model capability.
  • Four types, not two: working, episodic, semantic, and procedural. Procedural memory is the one teams forget they have.
  • Keep short-term and long-term memory separate. Appending everything to history is an overflow, not memory.
  • Sliding window plus rolling summary handles most conversational cases. A profile store handles most cross-session ones.
  • Store stable, timestamped, attributed facts. Prefer confirmed values over model-inferred ones.
  • Make memory visible, correctable, and deletable, and keep secrets out of it entirely.
  • Retrieved memory is untrusted input and can be stale. Re-confirm anything consequential.

Next: Hallucination & Groundedness - everything so far feeds the model context; this is what happens when the context is not enough.