Context Windows

The context window is the total number of tokens a model can process in one call, input and output combined. It is the hardest constraint in an LLM application: exceed it and the call fails or silently loses information.

A large window removes the failure but not the design problem. Filling 200K tokens on every call is slow, expensive, and measurably less accurate than sending 8K of well-chosen context.

For how tokens are counted and billed, see Tokens. This page is about managing what goes in the window.


What Fills It

Every component below competes for the same budget. Only one of them is the user's actual question.

One Context Window
A fixed token budget, shared by all of these
System Prompt
Fixed, paid every call
Tools
Tool definitions: grows with every tool registered
History
Grows every turn
Retrieved Docs
Grows with top_k
User Question
The only part anyone asked for
Output Space
Must be reserved, not discovered
ComponentTypical sizeGrowth behaviour
System prompt500 to 2,000 tokensFixed, but paid on every call
Tool and function definitions100 to 500 tokens per toolGrows with every tool you register
Retrieved documents500 to 2,000 tokens per chunkGrows with top_k
Conversation historyUnboundedGrows with every turn
User question50 to 500 tokensFixed per call
Reserved output space200 to 2,000 tokensMust be reserved, not discovered

The last row is the one most often missed. Input and output share the window, so a request that fills the window with input leaves no room for a response and errors on generation rather than on submission.


In Practice: A Long RAG Conversation

The failure is easier to see with numbers on it. A policy assistant on a 128,000-token model, twenty turns into a conversation:

What is in the windowTokens
System prompt and tool definitions2,500
Twenty turns of history, questions and full answers61,000
Ten retrieved policy chunks at top_k=1018,000
The current question200
Reserved for the answer2,000
Total83,700

That fits, so nothing errors. The problem is what it does to quality. The original question that set the task up is now 60,000 tokens back, sitting in the middle of the window where recall is weakest, and the ten chunks retrieved for this turn are competing with nineteen turns of answers that already quoted other chunks. The assistant starts blending policies, or answers the question it was asked five turns ago.

Nothing in the logs says so. The fix is not a bigger window, it is deciding what to drop: summarise history older than five turns, drop top_k to three with a re-ranker, and re-state the original task in the system prompt so it is never the thing that ages out.


What Happens on Overflow

Behaviour differs by provider and by how your own code is written, which is why overflow tends to surface as a confusing production bug rather than a clean error.

Where the limit is hitSymptom
Provider API rejects the requestHard error, easy to detect and handle
Your framework truncates historyModel silently forgets earlier turns and starts contradicting itself
Your framework truncates retrieved chunksAnswer quality drops with no error and no log line
Output space exhausted mid-generationResponse cuts off mid-sentence or returns invalid JSON

Count tokens before the call and decide what to drop deliberately. Letting a framework default make that choice is how a system loses the one chunk that contained the answer.


Position Matters

Recall is not uniform across the window. Models attend most reliably to the beginning and end of their input, and least reliably to the middle. A fact placed halfway through a long context is materially more likely to be missed than the same fact at either edge.

Start
System prompt, task instructions, output schema
Middle
Bulk retrieved documents. Weakest recall
End
Highest-ranked chunks, user question, reminders

Practical consequences for prompt layout:

  • Put instructions and the output schema first, and repeat the critical constraint immediately before the question.
  • Put the highest-scoring retrieved chunk last, not first. Retrieval ranking and prompt ordering should point the same way.
  • Put the user's question at the very end, after the retrieved context.

Management Strategies

StrategyHow it worksCost of using it
Sliding windowKeep the last N turns, drop the restLoses early context with no warning to the user
Rolling summaryReplace dropped turns with a generated summaryOne extra model call, and summary drift over long sessions
RerankingRetrieve broadly, then rank and keep the top few chunksAdds a reranker call, usually the best accuracy-per-token trade available
Chunk right-sizingMatch chunk size to question type rather than a fixed 1,000 tokensRequires evaluation to tune, not a one-off decision
Structured compressionSend extracted fields instead of raw source textExtraction step can drop something the model needed
Prompt cachingProvider caches the stable prefix across callsOnly helps if the prefix is genuinely stable, so order the prompt static-first

Prompt caching interacts directly with prompt layout. Put the system prompt and tool definitions first and keep them byte-identical between calls, because anything that changes early invalidates the cache for everything after it.


Running example: NLQ

For "What was revenue last quarter by region?" the schema is the budget problem. A warehouse with hundreds of tables will not fit in the window, and sending all of it would bury the few tables that matter. The fix is retrieval over table and column descriptions, so only the relevant schema goes in. See Text-to-SQL.

Sizing the Budget

Work backwards from the window rather than forwards from the content.

Model window                    128,000 tokens
- Reserved output                 2,000
- System prompt + tools           2,500
- Conversation history (capped)   8,000
= Available for retrieval       115,500 tokens

Then cap retrieval well below what is available. Available space is not a target. If 6 chunks answer the question as well as 40, sending 40 costs more, responds slower, and buries the useful chunk in the weakest part of the window.


Key Takeaways

  • Input and output share the window. Reserve output space explicitly or generation fails after you have already paid for the input.
  • Overflow often manifests as silent truncation rather than an error. Count tokens yourself and choose what to drop.
  • Recall is weakest in the middle of the context. Instructions first, best chunk and the question last.
  • More context is not better context. Retrieve broadly, rerank, then send few.
  • Keep the prompt prefix stable and static-first so provider prompt caching can apply to it.

Next: Prompting - once you know what fits in the window, the prompt is how you use it.