AI Play Book

Large Language Models

An LLM is a model trained on a very large text corpus until it can predict what text comes next. Everything it appears to do - answering, summarising, translating, writing code - is that one operation applied repeatedly.

Treating it as anything more than that leads to the two most common design mistakes: assuming it knows your business data, and assuming it will give the same answer twice.

How It Produces Text

The model receives a sequence of tokens and returns a probability distribution over every token in its vocabulary. One token is selected, appended to the sequence, and the process repeats until a stop condition is hit.

Prompt: "The invoice total is"

Step 1 → candidates: [" $" 0.41] [" the" 0.12] [" calculated" 0.07] ...  → " $"
Step 2 → candidates: ["1" 0.22] ["4" 0.18] ["2" 0.15] ...                → "1"
Step 3 → ...

Two parameters control the selection, and they are the levers that matter in production:

Parameter	What it does	When to use
`temperature`	Flattens or sharpens the distribution before sampling. 0 picks the highest-probability token every time.	Use 0 for extraction, classification, and routing. Raise it only for drafting and ideation.
`top_p`	Samples only from the smallest set of tokens whose probabilities sum to `p`.	Leave at default and tune `temperature` instead. Tuning both at once makes results hard to reason about.

temperature: 0 reduces variance but does not eliminate it. Batching, model updates, and floating-point non-determinism on GPU still produce occasional differences, so do not build a system that requires byte-identical output across runs.

In practice both parameters are set on the call alongside the schema that constrains the output:

# Extraction: deterministic sampling, schema-enforced output
import json
from openai import AzureOpenAI

client = AzureOpenAI(...)

response = client.chat.completions.create(
    model="gpt-4o",
    temperature=0,                 # take the top token at every step
    messages=[
        {"role": "system", "content": "Extract invoice fields. Return JSON only."},
        {"role": "user", "content": invoice_text},
    ],
    response_format={
        "type": "json_schema",
        "json_schema": {
            "name": "invoice",
            "strict": True,
            "schema": {
                "type": "object",
                "properties": {
                    "invoice_number": {"type": "string"},
                    "total": {"type": "number"},
                    "currency": {"type": "string"},
                },
                "required": ["invoice_number", "total", "currency"],
                "additionalProperties": False,
            },
        },
    },
)

data = json.loads(response.choices[0].message.content)
print(response.usage.total_tokens)   # the number that drives cost

Three things in that call matter more than the model name. temperature=0 removes sampling variance. strict: True makes the provider enforce the schema rather than asking the model to honour it in prose, which is the difference between a parse error once a week and once an hour. And usage.total_tokens is the figure to log from day one, because it is what the bill is calculated from. See Tokens for how that number is built up.

Foundation, Fine-Tuned, and Task Models

These three get used interchangeably in scoping conversations and shouldn't be.

Type	What it is	When to use
Foundation model	Pretrained on broad data, adapted per task through prompting alone. GPT, Claude, Gemini, Llama.	Almost always the starting point. Prompting plus retrieval solves most problems without touching model weights.
Fine-tuned model	A foundation model further trained on a task-specific dataset to shift its behaviour, format, or domain vocabulary.	Consistent output format, a domain register prompting cannot reach, or cost reduction by moving a task onto a smaller model. See Fine-Tuning.
Task-specific model	Trained for one narrow job: embeddings, reranking, OCR, classification, speech.	When the job is narrow. A dedicated embedding or OCR model beats a general LLM on cost, latency, and accuracy for its task.

The order to try things in is prompting, then retrieval, then fine-tuning. Fine-tuning first is the expensive way to discover the prompt was the problem.

The Training Cutoff

A model's parameters are frozen at training time. It has no awareness of your systems, today's date, or anything published after its cutoff, and it will not tell you when a question falls outside what it knows.

Everything current, private, or verifiable arrives through the pink path, not the grey one. That path is what RAG and tool calling exist to build.

Sizing a Model to a Task

Model choice is a cost and latency decision far more often than a capability one. The largest model in a family is typically 15 to 30 times the price of the smallest, and the smallest handles a surprising share of production work.

Task	Size class	Why
Classification, routing, tagging	Small	Constrained output space. A large model adds cost, not accuracy.
Field extraction from a document	Small to mid	Accuracy comes from the prompt and the input quality, not model scale.
Summarisation, rewriting, drafting	Mid	Needs fluency and instruction-following, not deep reasoning.
Multi-step reasoning, planning, code generation	Large	Error compounds across steps, so per-step quality dominates cost.
Agentic loops with tool calls	Large for the planner, small for the workers	Reasoning about which tool to call is the hard part. Executing a formatting step is not.

Route by task rather than standardising on one model. A router that sends classification to a small model and planning to a large one usually cuts inference spend by more than half without a measurable quality change. Measure that claim on your own evaluation set before committing to it.

Key Takeaways