Fine-tuning
Fine-tuning continues training a base model on your own examples, adjusting its weights so it responds the way your examples respond without being told to in the prompt. The output is a private model variant you then have to host, version, and maintain.
The rule that decides whether fine-tuning is the right tool: fine-tuning teaches behaviour, format, and style. RAG supplies knowledge. A fine-tuned model learns how to answer. It does not reliably learn what is true this quarter.
Most failed fine-tuning projects start when a team tries to push facts into the weights - product catalogues, pricing tables, policy documents, customer records. Those facts go stale the moment the source system changes, they cannot be cited back to a source, and correcting a single wrong one means retraining the whole model. RAG solves that problem directly by fetching the facts at query time, and it is almost always the cheaper first move.
If the problem you are solving is "the model does not know our data", fine-tuning is the wrong tool. Fine-tune to change how the model writes and structures its answers. Retrieve to change what it knows.
The Decision Framework
Work down this table in order. Every step is cheaper, faster to iterate on, and easier to reverse than the one below it. Skipping straight to fine-tuning is the most common and most expensive mistake in this space.
| Approach | What it changes | Iteration speed | Use when |
|---|---|---|---|
| Prompt engineering | Instructions and few-shot examples only | Minutes | Always start here. General tasks, no training data, requirements still moving |
| RAG | What the model can see at query time | Hours | The gap is knowledge. Content changes, answers need citations, access control matters |
| Fine-tuning | The model weights | Days to weeks | The gap is behaviour. Prompting has plateaued and you have curated examples |
| RAG + fine-tuning | Both | Days to weeks | Domain writing style or output shape and grounding in current documents |
Is the model failing because it lacks information, or because it has the information and still responds in the wrong shape, tone, or level of detail? The first is a retrieval problem. Only the second is a fine-tuning problem.
When Fine-tuning Genuinely Wins
- Consistent structured output. The model must emit the same JSON schema, XML shape, or field ordering on every call. Prompting gets this mostly right; fine-tuning gets it right often enough to remove the retry loop and the repair parser.
- A domain tone or format that resists prompting. Regulatory summaries, clinical notes, legal drafting conventions, an internal report house style. When the instruction to reproduce a style is longer than the answer itself, the style belongs in the weights.
- Latency and cost reduction on a narrow task. A small fine-tuned model can match a much larger general model on one specific job, at a fraction of the per-call cost and latency. On high-volume tasks this is the strongest financial case for fine-tuning.
- Prompt compression. Fine-tuning moves a long system prompt into the weights. A system prompt that runs on every call is a recurring token bill, as covered in Tokens. Removing 1,500 tokens of instructions from every request compounds quickly at volume.
- Classification and extraction at scale. Fixed label sets, high call volume, and clear ground truth. This is the most reliable fine-tuning use case in production, and usually the easiest to evaluate.
When Fine-tuning Is the Wrong Tool
- The knowledge changes. Anything with a refresh cycle - prices, inventory, policies, staff, documentation - belongs in a retrieval index, not in weights. Retraining is not an update mechanism.
- Answers need citations. A fine-tuned model cannot tell you where an answer came from. If a user, an auditor, or a regulator needs the source, you need retrieval.
- The dataset is small or dirty. Fine-tuning on a few dozen inconsistent examples reliably makes the model worse, and worse in a confident, hard-to-debug way. It learns your labelling mistakes as faithfully as it learns the task.
- Prompting has not actually been tried. "We tried a prompt and it did not work" usually means one prompt, no few-shot examples, and no structured output constraints. Exhaust that first.
- Retrieval quality is the real problem. If the model is given the wrong chunks, no amount of weight adjustment fixes the answer. Debug the retrieval pipeline first.
- Requirements are still moving. Every requirement change invalidates the training set. Fine-tune when the task has stabilised.
Training Data Preparation
This is the real cost centre. Compute is usually the cheapest part of a fine-tuning project; assembling, cleaning, and labelling examples is where most of the effort and nearly all of the failure lives.
The standard format is instruction and response pairs, one JSON object per line, using the chat message structure:
{"messages": [{"role": "system", "content": "Extract contract terms as JSON with keys: party, effective_date, termination_notice_days, governing_law."}, {"role": "user", "content": "This Agreement is entered into by Northwind Traders on 3 March 2025. Either party may terminate on ninety days written notice. Governed by the laws of England and Wales."}, {"role": "assistant", "content": "{\"party\": \"Northwind Traders\", \"effective_date\": \"2025-03-03\", \"termination_notice_days\": 90, \"governing_law\": \"England and Wales\"}"}]}
{"messages": [{"role": "system", "content": "Extract contract terms as JSON with keys: party, effective_date, termination_notice_days, governing_law."}, {"role": "user", "content": "Contoso Ltd and the supplier agree this MSA commences 1 January 2026. Termination requires 30 days notice in writing. New York law applies."}, {"role": "assistant", "content": "{\"party\": \"Contoso Ltd\", \"effective_date\": \"2026-01-01\", \"termination_notice_days\": 30, \"governing_law\": \"New York\"}"}]}
Rules that hold regardless of platform:
- Quality beats quantity, decisively. A few hundred clean, consistent examples outperform thousands of scraped or auto-generated ones. Inconsistent labelling across examples is the single most common cause of a tune that scores worse than the base model.
- Volume depends on the task. Narrow classification and fixed-format extraction need the least. Tone and style transfer need more. Treat published minimums as a floor for the mechanics to run, not a target for good results.
- Keep the system prompt identical across training records and at inference. A mismatch between the two is a frequent and easily missed source of degraded output.
- Hold out a validation split before training, typically 10-20% of examples, and keep a separate test set the training process never sees. The test set is what you score the baseline on later.
- Make the split representative. Random splits leak near-duplicates across train and test and produce flattering scores. Split by document, customer, or time period instead.
- Include the hard cases. Edge cases, ambiguous inputs, and the failure modes you actually saw in production. Training only on easy examples teaches the model the easy task.
The Workflow
Step 2 is the one teams skip, and skipping it makes the whole exercise unfalsifiable.
Score the un-tuned base model on your test set before you train, and record the number. Without that baseline you cannot tell whether the fine-tune improved anything, made no difference, or made things worse. A tuned model that "seems better" in a demo is not evidence.
Use task-appropriate metrics rather than training loss: exact-match or schema-validity rate for structured extraction, accuracy and per-class recall for classification, and a rubric-scored LLM-as-judge run for style and tone work. Compare the tuned model against three references - the base model with the same prompt, the base model with a better prompt, and a larger model with the same prompt. If a stronger prompt on the base model closes most of the gap, ship the prompt.
Two training approaches are worth distinguishing. Full fine-tuning updates all weights and needs substantial GPU capacity. Parameter-efficient methods such as LoRA train a small set of adapter weights on top of a frozen base, which is far cheaper, much faster to iterate on, and produces artefacts small enough to keep several variants around. LoRA is the sensible default for most enterprise work; reach for full fine-tuning only when adapters have been tried and measured.
Cost and Maintenance
Training cost is one-off and usually modest. The costs that get underestimated are the ones that recur.
| Cost | Shape | Notes |
|---|---|---|
| Data preparation | One-off, large | Human curation and labelling. Usually the dominant cost of the whole project |
| Training | One-off per run, but repeated | Rarely a single run. Budget for several attempts before one beats the baseline |
| Hosting | Ongoing | A custom model is typically billed on dedicated compute rather than per token. It costs money while idle |
| Evaluation | Ongoing | The test set has to be maintained and rerun, not written once |
| Re-tuning | Ongoing, unpredictable | Triggered by base model upgrades, data drift, or requirement changes |
The last row is the one that surprises people. A fine-tune is bound to a specific base model version. When the provider ships a better base model, your tuned variant does not inherit the improvement, and the new base model may already outperform your tune out of the box. Every upgrade forces the same decision: retrain on the new base, or retire the tune. Teams that treat fine-tuning as a one-time project rather than an owned asset with a maintenance budget end up running a stale model against a moving frontier.
The hosting economics also invert at low volume. Dedicated capacity for a fine-tuned model bills continuously, while a shared base model with a longer prompt bills only per call. Below a certain call volume, the more expensive prompt is the cheaper system.
Platform Support
| Platform | Fine-tuning position | Where to read more |
|---|---|---|
| Databricks | Strongest option for open-weight models. Fine-tune Llama, Mistral, and similar on your own data, with MLflow tracking and Unity Catalog governance. Requires GPU capacity and real MLOps discipline - it is not a one-click operation | Databricks |
| Azure OpenAI / Microsoft Foundry | Managed fine-tuning for hosted GPT models. Lowest operational overhead, least control over the training process, and you are tied to the provider's base model lifecycle | Foundry Models |
| Snowflake | Limited. Cortex is strongest where AI stays close to SQL; fine-tuning is among its weakest areas. Plan for external tooling or a handoff if custom training is on the roadmap | Snowflake |
The choice usually follows the model, not the platform. Open-weight models mean Databricks or your own GPU infrastructure. Hosted frontier models mean the provider's managed tuning service, with whatever constraints that implies.
Key Takeaways
- Fine-tuning teaches behaviour, format, and style. It does not reliably teach facts. Use RAG for knowledge that changes or needs citations.
- Exhaust prompt engineering and retrieval quality first. Most requests for fine-tuning are prompting or retrieval problems that have not been debugged properly.
- The strongest cases are consistent structured output, a domain style that resists prompting, and making a small model match a larger one on a narrow, high-volume task.
- Data preparation is the real cost and the real risk. A few hundred clean, consistently labelled examples beat thousands of noisy ones, every time.
- Score the un-tuned base model on your test set before training. Without that baseline, you cannot claim the fine-tune helped.
- Training is one-off; hosting, evaluation, and re-tuning are not. Every base model upgrade forces a retrain-or-retire decision, and that recurring cost is what teams underestimate.