Tokens
A token is a small piece of text that an AI model reads and understands. Every interaction with an LLM is measured, billed, and constrained in tokens.

What is a Token?
Text is not fed into a model word by word or character by character - it is split into tokens first. Tokens are sub-word units that balance vocabulary coverage with model efficiency.
"Artificial Intelligence is changing the world."
→ [Artificial] [Intelligence] [is] [changing] [the] [world] [.]
1012 29871 374 9123 279 6891 13
Rules of thumb:
- 1 token ≈ 4 characters of English text
- 1 token ≈ 0.75 words
- 1,000 tokens ≈ 750 words ≈ ~1.5 pages of text
Numbers, punctuation, and non-English text often tokenize less efficiently - the same 100 words in French or German may use more tokens than in English.
How Tokens Are Used in a Prompt
Every LLM call consumes tokens across all inputs combined:
All four components of the input count toward your token bill and context window usage.
Tokens and Cost
You are billed on both input and output tokens - separately.
| Component | What it includes | Billed as |
|---|---|---|
| Input tokens | System prompt + conversation history + user question + retrieved documents | Input rate |
| Output tokens | The model's generated response | Output rate (typically 3-4x input rate) |
Example (GPT-4o):
Input: 5,000 tokens × $2.50 / 1M = $0.0125
Output: 1,000 tokens × $10.00 / 1M = $0.010
Total cost per call: $0.0225
At 10,000 calls/day: $225/day → $6,750/month just on inference.
Why this matters for system design:
- A long system prompt runs on every call - 2,000 token system prompt × 10,000 calls/day = 20M tokens/day in system prompt alone
- Conversation history grows with each turn - unbounded history is a cost and context window problem
- RAG chunks add to input tokens - larger chunks = better recall but higher cost per query
Context Window
Tokens are also what the context window is measured in: system prompt, history, retrieved documents, the question, and the answer all draw on one fixed budget per call. Context Windows covers what fills it, what happens on overflow, and how to manage it.
Cost Optimization Strategies
| Strategy | How it reduces tokens |
|---|---|
| Concise system prompts | Fewer tokens on every single call |
| Sliding window history | Keep last N turns, not full history |
| Summarise old history | Replace old turns with a short summary |
| Right-size RAG chunks | Match chunk size to query complexity |
max_tokens parameter | Cap output length to prevent runaway responses |
| Model routing | Use smaller, cheaper models for simple tasks (classification, extraction) |
| Semantic caching | Return cached response for near-duplicate queries |
Key Takeaways
- Every character you send to an LLM costs money - system prompts, history, and RAG chunks all bill at the input rate.
- Output tokens cost 3-4x more than input tokens on most models. Keep responses concise.
- The context window is a hard limit - design conversation history and RAG retrieval to fit within it, not just to maximise recall.
- A 2,000 token system prompt running on 10,000 calls/day consumes 20M input tokens daily. Small prompt improvements compound significantly at scale.
- Use smaller models for simple tasks. GPT-4o-mini or Phi-4 at 10% of GPT-5 cost handles classification, extraction, and routing well.
Next: Context Windows - tokens add up to a fixed budget per call, and what you spend it on decides the answer.