Context Windows
The context window is the total number of tokens a model can process in one call, input and output combined. It is the hardest constraint in an LLM application: exceed it and the call fails or silently loses information.
A large window removes the failure but not the design problem. Filling 200K tokens on every call is slow, expensive, and measurably less accurate than sending 8K of well-chosen context.
For how tokens are counted and billed, see Tokens. This page is about managing what goes in the window.
What Fills It
Every component below competes for the same budget. Only one of them is the user's actual question.
| Component | Typical size | Growth behaviour |
|---|---|---|
| System prompt | 500 to 2,000 tokens | Fixed, but paid on every call |
| Tool and function definitions | 100 to 500 tokens per tool | Grows with every tool you register |
| Retrieved documents | 500 to 2,000 tokens per chunk | Grows with top_k |
| Conversation history | Unbounded | Grows with every turn |
| User question | 50 to 500 tokens | Fixed per call |
| Reserved output space | 200 to 2,000 tokens | Must be reserved, not discovered |
The last row is the one most often missed. Input and output share the window, so a request that fills the window with input leaves no room for a response and errors on generation rather than on submission.
What Happens on Overflow
Behaviour differs by provider and by how your own code is written, which is why overflow tends to surface as a confusing production bug rather than a clean error.
| Where the limit is hit | Symptom |
|---|---|
| Provider API rejects the request | Hard error, easy to detect and handle |
| Your framework truncates history | Model silently forgets earlier turns and starts contradicting itself |
| Your framework truncates retrieved chunks | Answer quality drops with no error and no log line |
| Output space exhausted mid-generation | Response cuts off mid-sentence or returns invalid JSON |
Count tokens before the call and decide what to drop deliberately. Letting a framework default make that choice is how a system loses the one chunk that contained the answer.
Position Matters
Recall is not uniform across the window. Models attend most reliably to the beginning and end of their input, and least reliably to the middle. A fact placed halfway through a long context is materially more likely to be missed than the same fact at either edge.
Practical consequences for prompt layout:
- Put instructions and the output schema first, and repeat the critical constraint immediately before the question.
- Put the highest-scoring retrieved chunk last, not first. Retrieval ranking and prompt ordering should point the same way.
- Put the user's question at the very end, after the retrieved context.
Management Strategies
| Strategy | How it works | Cost of using it |
|---|---|---|
| Sliding window | Keep the last N turns, drop the rest | Loses early context with no warning to the user |
| Rolling summary | Replace dropped turns with a generated summary | One extra model call, and summary drift over long sessions |
| Reranking | Retrieve broadly, then rank and keep the top few chunks | Adds a reranker call, usually the best accuracy-per-token trade available |
| Chunk right-sizing | Match chunk size to question type rather than a fixed 1,000 tokens | Requires evaluation to tune, not a one-off decision |
| Structured compression | Send extracted fields instead of raw source text | Extraction step can drop something the model needed |
| Prompt caching | Provider caches the stable prefix across calls | Only helps if the prefix is genuinely stable, so order the prompt static-first |
Prompt caching interacts directly with prompt layout. Put the system prompt and tool definitions first and keep them byte-identical between calls, because anything that changes early invalidates the cache for everything after it.
Sizing the Budget
Work backwards from the window rather than forwards from the content.
Model window 128,000 tokens
- Reserved output 2,000
- System prompt + tools 2,500
- Conversation history (capped) 8,000
= Available for retrieval 115,500 tokens
Then cap retrieval well below what is available. Available space is not a target. If 6 chunks answer the question as well as 40, sending 40 costs more, responds slower, and buries the useful chunk in the weakest part of the window.
Key Takeaways
- Input and output share the window. Reserve output space explicitly or generation fails after you have already paid for the input.
- Overflow often manifests as silent truncation rather than an error. Count tokens yourself and choose what to drop.
- Recall is weakest in the middle of the context. Instructions first, best chunk and the question last.
- More context is not better context. Retrieve broadly, rerank, then send few.
- Keep the prompt prefix stable and static-first so provider prompt caching can apply to it.