Tokens
A token is a small piece of text that an AI model reads and understands. Every interaction with an LLM is measured, billed, and constrained in tokens.
What is a Token?
Text is not fed into a model word by word or character by character - it is split into tokens first. Tokens are sub-word units that balance vocabulary coverage with model efficiency.
"Artificial Intelligence is changing the world."
→ [Artificial] [Intelligence] [is] [changing] [the] [world] [.]
1012 29871 374 9123 279 6891 13
Rules of thumb:
- 1 token ≈ 4 characters of English text
- 1 token ≈ 0.75 words
- 1,000 tokens ≈ 750 words ≈ ~1.5 pages of text
Numbers, punctuation, and non-English text often tokenize less efficiently - the same 100 words in French or German may use more tokens than in English.
How Tokens Are Used in a Prompt
Every LLM call consumes tokens across all inputs combined:
All four components of the input count toward your token bill and context window usage.
Tokens and Cost
You are billed on both input and output tokens - separately.
| Component | What it includes | Billed as |
|---|---|---|
| Input tokens | System prompt + conversation history + user question + retrieved documents | Input rate |
| Output tokens | The model's generated response | Output rate (typically 3-4x input rate) |
Example (GPT-4o):
Input: 5,000 tokens × $2.50 / 1M = $0.0125
Output: 1,000 tokens × $10.00 / 1M = $0.010
Total cost per call: $0.0225
At 10,000 calls/day: $225/day → $6,750/month just on inference.
Why this matters for system design:
- A long system prompt runs on every call - 2,000 token system prompt × 10,000 calls/day = 20M tokens/day in system prompt alone
- Conversation history grows with each turn - unbounded history is a cost and context window problem
- RAG chunks add to input tokens - larger chunks = better recall but higher cost per query
Context Window
The context window is the total number of tokens an LLM can process in a single call - input + output combined.
| Model | Context Window |
|---|---|
| GPT-5.5 | 256K tokens |
| GPT-5 | 272K tokens |
| GPT-4o | 128K tokens |
| Claude Opus 5 | 200K tokens |
| Llama 4 | 128K tokens |
What fills the context window:
System prompt ~500-2,000 tokens
Conversation history grows with turns
Retrieved documents ~500-2,000 tokens per chunk × number of chunks
User question ~50-500 tokens
Response ~200-2,000 tokens
When the total exceeds the context window, the model either errors or truncates - usually conversation history is dropped first. See Context Windows for overflow behaviour, position effects, and the management strategies.
Practical limit: even with a 128K context window, sending 128K tokens on every call is expensive and slow. Design prompts to use context windows efficiently, not just fit within them.
Cost Optimization Strategies
| Strategy | How it reduces tokens |
|---|---|
| Concise system prompts | Fewer tokens on every single call |
| Sliding window history | Keep last N turns, not full history |
| Summarise old history | Replace old turns with a short summary |
| Right-size RAG chunks | Match chunk size to query complexity |
max_tokens parameter | Cap output length to prevent runaway responses |
| Model routing | Use smaller, cheaper models for simple tasks (classification, extraction) |
| Semantic caching | Return cached response for near-duplicate queries |
Key Takeaways
- Every character you send to an LLM costs money - system prompts, history, and RAG chunks all bill at the input rate.
- Output tokens cost 3-4x more than input tokens on most models. Keep responses concise.
- The context window is a hard limit - design conversation history and RAG retrieval to fit within it, not just to maximise recall.
- A 2,000 token system prompt running on 10,000 calls/day consumes 20M input tokens daily. Small prompt improvements compound significantly at scale.
- Use smaller models for simple tasks. GPT-4o-mini or Phi-4 at 10% of GPT-5 cost handles classification, extraction, and routing well.