Tokens

A token is a small piece of text that an AI model reads and understands. Every interaction with an LLM is measured, billed, and constrained in tokens.

What is a Token?


What is a Token?

Text is not fed into a model word by word or character by character - it is split into tokens first. Tokens are sub-word units that balance vocabulary coverage with model efficiency.

"Artificial Intelligence is changing the world."

→ [Artificial] [Intelligence] [is] [changing] [the] [world] [.]
     1012        29871        374    9123       279    6891   13

Rules of thumb:

  • 1 token ≈ 4 characters of English text
  • 1 token ≈ 0.75 words
  • 1,000 tokens ≈ 750 words ≈ ~1.5 pages of text

Numbers, punctuation, and non-English text often tokenize less efficiently - the same 100 words in French or German may use more tokens than in English.


How Tokens Are Used in a Prompt

Every LLM call consumes tokens across all inputs combined:

Input
Every part is billed
System Prompt
History
User Question
Retrieved Docs
From RAG
Model
LLM
Output
Also billed
Response
Generated text

All four components of the input count toward your token bill and context window usage.


Tokens and Cost

You are billed on both input and output tokens - separately.

ComponentWhat it includesBilled as
Input tokensSystem prompt + conversation history + user question + retrieved documentsInput rate
Output tokensThe model's generated responseOutput rate (typically 3-4x input rate)

Example (GPT-4o):

Input:  5,000 tokens × $2.50 / 1M = $0.0125
Output: 1,000 tokens × $10.00 / 1M = $0.010
Total cost per call:                  $0.0225

At 10,000 calls/day: $225/day → $6,750/month just on inference.

Why this matters for system design:

  • A long system prompt runs on every call - 2,000 token system prompt × 10,000 calls/day = 20M tokens/day in system prompt alone
  • Conversation history grows with each turn - unbounded history is a cost and context window problem
  • RAG chunks add to input tokens - larger chunks = better recall but higher cost per query

Context Window

Tokens are also what the context window is measured in: system prompt, history, retrieved documents, the question, and the answer all draw on one fixed budget per call. Context Windows covers what fills it, what happens on overflow, and how to manage it.


Cost Optimization Strategies

StrategyHow it reduces tokens
Concise system promptsFewer tokens on every single call
Sliding window historyKeep last N turns, not full history
Summarise old historyReplace old turns with a short summary
Right-size RAG chunksMatch chunk size to query complexity
max_tokens parameterCap output length to prevent runaway responses
Model routingUse smaller, cheaper models for simple tasks (classification, extraction)
Semantic cachingReturn cached response for near-duplicate queries

Key Takeaways

  • Every character you send to an LLM costs money - system prompts, history, and RAG chunks all bill at the input rate.
  • Output tokens cost 3-4x more than input tokens on most models. Keep responses concise.
  • The context window is a hard limit - design conversation history and RAG retrieval to fit within it, not just to maximise recall.
  • A 2,000 token system prompt running on 10,000 calls/day consumes 20M input tokens daily. Small prompt improvements compound significantly at scale.
  • Use smaller models for simple tasks. GPT-4o-mini or Phi-4 at 10% of GPT-5 cost handles classification, extraction, and routing well.

Next: Context Windows - tokens add up to a fixed budget per call, and what you spend it on decides the answer.