# Fine-Tuning vs Prompting, Plainly
> When training your own model is - and isn't - worth it: prompting steers at request time, RAG adds knowledge, fine-tuning changes the model's default behavior, and the no-nonsense order is to try them cheapest-first.
---
# Fine-Tuning vs Prompting, Plainly
At some point - usually after a demo goes well and someone with a budget gets excited - the question lands
on your desk: *"Should we fine-tune our own model?"* It sounds like the serious, grown-up answer. It sounds
like what real AI teams do. And it is, sometimes, exactly the wrong move that costs a quarter and ships
nothing.
The plain truth is that most teams reaching for fine-tuning didn't need it. They needed a better prompt, or
they needed to feed the model the right documents at request time. Fine-tuning is a real tool with a real
job - but it's the most expensive way to steer a model, it locks you in the hardest, and it's the one people
reach for first for the wrong reasons.
This guide gives you the mental model to tell the three approaches apart, a clear-eyed look at what fine-tuning
actually costs, and a decision order you can defend in a meeting. This is the capstone of the AI/ML track -
it assumes you've met prompting, RAG, and calling an LLM API in the sibling guides, and pulls them together
into one decision.
## How to read this
- **Need to decide right now?** Jump to [Phase 3: Choosing - the No-Nonsense Order](03-choosing-the-order.md)
and use the decision table at the top.
- **Want it to finally make sense?** Read in order - each phase builds on the last. Phase 1 gives you the
three-way mental model, Phase 2 shows what fine-tuning really involves, and Phase 3 turns it into a
decision.
## The phases
1. **[Three Ways to Steer a Model](01-three-ways-to-steer-a-model.md)** - the mental model: prompting changes
the *instructions*, RAG changes the *knowledge*, fine-tuning changes the *behavior*. The distinction that
the whole decision rests on.
2. **[What Fine-Tuning Actually Involves](02-what-fine-tuning-actually-involves.md)** - the dataset (where the
real cost lives), the training run, hosting your tuned model, the lighter LoRA approach, and how you'd
know if it worked.
3. **[Choosing - the No-Nonsense Order](03-choosing-the-order.md)** - try prompt → RAG → fine-tune, in that
order, because each step costs more and locks you in more. A decision table, and the two traps that catch
everyone.
> Deeper material - building a training pipeline, distillation, RLHF, and serving infrastructure at scale -
> is deliberately out of scope here. This guide is about the *decision*, not the implementation. Once you've
> clearly decided fine-tuning is right, your model provider's tuning docs are your next stop.
**Related guides:** [Prompt Engineering, Plainly](/guides/prompt-engineering-plainly) ·
[RAG, Explained](/guides/rag-explained) · [Using an LLM API](/guides/using-an-llm-api)
---
# Three Ways to Steer a Model
Out of the box, a large language model is a generalist. It will write you a sonnet, a SQL query, and a
breakup text with the same shrug of competence. Your job is almost never "use a model" - it's "make *this*
model reliably do *our specific thing*, in *our specific way*." That's steering.
There are exactly three levers you can pull, and the entire fine-tuning-vs-prompting debate comes down to
knowing which lever fixes which problem. Pull the wrong one and you'll spend weeks and a real budget solving
a problem the cheapest lever would have solved in an afternoon.
## The one picture to hold onto
A model's output depends on two things: the **weights** (the billions of numbers baked in during training -
the model's "instincts," its defaults) and the **context** (everything you hand it at request time - the
prompt, the conversation, any documents you paste in). Two of your three levers work on the context; one
works on the weights.
```mermaid
flowchart TD
subgraph RT["REQUEST TIME - cheap, instant, reversible"]
direction LR
P["PROMPTING changes WHAT you ask for"]
R["RAG changes WHAT facts it sees"]
end
subgraph TT["TRAINING TIME - expensive, slow, sticky"]
F["FINE-TUNING changes HOW it answers by default"]
end
P -->|fed in as context| OUT[model output]
R -->|fed in as context| OUT
F -->|baked into the weights| OUT
```
*The dividing line that matters most:* **RAG adds knowledge. Fine-tuning teaches behavior.** Almost every
expensive mistake in this space is someone fine-tuning to add knowledge (which RAG does better and cheaper)
or someone trying to prompt their way to a consistent format at scale (which is exactly fine-tuning's job).
## Lever 1 - Prompting: change the instructions at request time
**What it actually is.** You write the model better instructions - that's the whole lever. Tell it who to be
("You are a terse senior code reviewer"), what to do, what format to use, and show it a couple of examples
right there in the prompt. Nothing about the model changes; you're feeding a generalist a clearer brief each
time you call it.
**What it does in real life.** It's the fastest, cheapest, most reversible way to steer. Edit a string and
the behavior changes on the next request - no training, no waiting, no infrastructure. Modern models are
startlingly steerable this way.
**A real example.** Showing the model two examples inside the prompt ("few-shot" prompting) often gets you
most of the way to a consistent format:
```text
System: Classify each support ticket as BUG, BILLING, or FEATURE.
Reply with only the label.
User: "I was charged twice this month." → BILLING
User: "The export button does nothing." → BUG
User: "Can you add dark mode?" → FEATURE
User: "App crashes when I open settings." →
```
*What just happened:* You didn't change the model at all - you showed it the pattern in the context, and a
capable model will continue it (`BUG`). This is the lever to exhaust before considering the others; it's
covered properly in [Prompt Engineering, Plainly](/guides/prompt-engineering-plainly).
**The gotcha.** Prompting has a ceiling. Stuff enough rules and examples into every request and three things
creep up: the prompt gets long (you pay per token, every call), the model starts ignoring instructions
buried in the middle, and behavior stays *mostly* consistent rather than *reliably* consistent. When you hit
that ceiling for real - not in your imagination - that's the first real signal another lever might be
warranted.
## Lever 2 - RAG: inject knowledge at request time
**What it actually is.** RAG (Retrieval-Augmented Generation) means: before you call the model, go fetch the
relevant facts - from your docs, your database, your knowledge base - and paste them into the prompt. The
model answers using text you handed it a moment ago, instead of relying on whatever it happened to absorb
during training.
**What it does in real life.** It gives a general model *your* specific, current knowledge without changing
the model at all. Pricing page changed this morning? Update the document; the next answer is correct. A
customer asks about an internal policy the model has never seen? Retrieve the doc and hand it over - the
model reads it like an open-book exam.
**A real example.** Conceptually, RAG turns a closed-book question into an open-book one:
```text
Without RAG: "What's our refund window?" → model guesses from generic training → maybe wrong
With RAG: [retrieve: refund-policy.md]
"Using the policy below, what's our refund window?
--- Refunds accepted within 30 days of purchase. ---"
→ model reads it → "30 days." (correct, sourced)
```
*What just happened:* The model didn't *learn* your refund policy - it *read* it, at request time, from
context you supplied. Change the policy file and the answer changes immediately, nothing to retrain. The
full machinery (chunking, embeddings, retrieval) lives in [RAG, Explained](/guides/rag-explained).
**The gotcha.** RAG fixes *what the model knows*, not *how it behaves*. It will happily retrieve your refund
policy and then answer in a rambling, off-brand, wrong-format way, because RAG never touched its behavior.
If your problem is "the model doesn't know our facts," RAG is your tool. If it's "the model knows plenty but
won't consistently answer in our voice/format," RAG won't help - that's the doorway to the third lever.
## Lever 3 - Fine-tuning: change the model's default behavior
**What it actually is.** Fine-tuning takes an existing trained model and nudges its **weights** - the numbers
that *are* the model - by training it further on hundreds or thousands of your own example input/output
pairs. You're not giving it instructions or documents; you're changing its instincts so the behavior you
want becomes its *default*, with no special prompting required.
📝 **Weights.** The billions of numbers learned during a model's original training. They encode everything
the model "knows" and every habit it has. Prompting and RAG leave the weights untouched; fine-tuning is the
only lever that edits them.
**What it does in real life.** After fine-tuning on, say, three thousand examples of your support replies,
the model writes in your support voice *by default* - terse where you're terse, with your standard sign-off -
even from a bare prompt. The behavior is baked in, so prompts get shorter (the rules live in the weights now,
not in every request) and consistency gets tighter than prompting alone reliably delivers.
**A real example.** The shape of fine-tuning is a file of demonstrations:
```text
{"input": "where's my order #4821",
"output": "Hi! Order #4821 shipped Tuesday - tracking: . Anything else? - Support"}
{"input": "this is broken and I'm furious",
"output": "I'm really sorry. Let's fix this fast. Can you tell me what you were doing when it broke? - Support"}
... a few thousand more pairs in exactly the tone and format you want ...
```
*What just happened:* You're not telling the model rules in prose - you're showing it the behavior, over and
over, and training adjusts the weights until that behavior is the path of least resistance. The voice,
structure, and sign-off become *defaults*, not instructions you repeat every call.
**The gotcha - the one that costs the most.** Fine-tuning is a poor way to teach *facts*. People assume
"training it on our documents" will make the model *know* our documents. What it mostly learns is the
*style* and *shape* of those documents, not reliable recall of their contents - and any fact that changes
next week is now frozen into weights you'd have to retrain to update. For knowledge, RAG wins almost every
time. Hold this line: **fine-tune for behavior, retrieve for knowledge.**
**Why this saves you later.** When the budget-holder asks "should we train our own model so it knows our
product?", you have the clear answer ready: training won't reliably make it *know* the product (that's
RAG), and you almost certainly haven't exhausted prompting yet. You've just saved a quarter.
## Recap
1. A model's output comes from its **weights** (its defaults) plus its **context** (what you hand it per
request). Prompting and RAG work on context; fine-tuning works on weights.
2. **Prompting** changes the *instructions* at request time - cheapest, fastest, fully reversible, but it has
a ceiling.
3. **RAG** injects *knowledge* at request time - the right tool when the model doesn't know your facts, and
facts change without retraining.
4. **Fine-tuning** changes the model's *default behavior* by editing its weights - the right tool for
consistent voice/format/style, the wrong tool for teaching facts.
5. The line that drives every decision: **RAG adds knowledge, fine-tuning teaches behavior.**
Now that you can tell the levers apart, the next phase looks plainly at what pulling the fine-tuning lever
actually takes - because the cost is rarely where people expect it.
---
# What Fine-Tuning Actually Involves
If you've decided (or are tempted) to fine-tune, here's the part the demos skip: what it actually takes, day
to day, and where the cost really lives. Spoiler - it's almost never the training run. The training is a
button. The hard part is everything around it.
Knowing the real shape of the work is what stops you from saying "yes" in a meeting and discovering three
weeks later that the expensive part hadn't even started.
## The pipeline, end to end
Four stages, wildly unequal in effort:
```mermaid
flowchart LR
D["1. THE DATASET curated examples (where the real cost lives)"] --> T["2. TRAINING the run (mostly a button)"]
T --> H["3. HOSTING / SERVING (ongoing, not one-and-done)"]
H --> E["4. EVALUATION did it work? (skip it and you're flying blind)"]
```
The widest part of that diagram is stage 1, and that's not an accident.
## Stage 1 - The dataset (this is the real cost)
**What it actually is.** A fine-tuning dataset is a collection of example pairs: an input, and the *exact*
output you wish the model had produced. Not roughly - exactly. Each pair demonstrates the behavior you're
trying to make default. You'll typically want hundreds at a minimum and often thousands, and every one has
to be *good*.
**What it does in real life.** This is where weeks go. Someone who understands the domain - not a junior with
a spare afternoon - has to gather real inputs, write or clean up the ideal outputs, make them consistent with
each other, and check them. Inconsistent examples teach the model an inconsistent habit. The dataset *is* the
product; the model just absorbs whatever's in it.
⚠️ **The gotcha - garbage in, garbage model.** This is the single most important sentence in this guide:
**a fine-tuned model is only as good as its training data, and bad data is worse than no fine-tuning at all.**
If your examples contradict each other, contain mistakes, or aren't actually in the voice you want, you'll
get a model that has *confidently learned the wrong habit* - harder to debug than a base model, because the
bad behavior is now baked into the weights instead of sitting in a prompt you can edit. Curation is not the
boring prerequisite. Curation is the job.
**Why this saves you later.** Scope the *dataset* first, plainly. If nobody on the team has the time or
domain knowledge to produce a few thousand consistent, high-quality examples, you don't have a fine-tuning
project yet - you have a data project wearing a fine-tuning costume. Better to learn that on day one.
## Stage 2 - The training run
**What it actually is.** You hand your dataset to a training process - through your model provider's
fine-tuning API, or your own setup if you're hosting open-weight models - and it adjusts the weights over a
number of passes through your data. With a hosted provider, this is genuinely close to "upload file, click
start, wait."
**What it does in real life.** It runs for a while (minutes to hours) and hands you back a tuned model you
can call like any other. The compute has a real but usually modest cost next to stage 1's human cost - this
is the stage people *picture* when they think "fine-tuning," and it's the easiest by far.
**The gotcha.** A training run that completes without errors is not a training run that succeeded. "It
finished" tells you nothing about whether the model got *better* - only that the machinery ran. Two failure
modes to know by name:
📝 **Overfitting.** The model memorizes your training examples instead of learning the general behavior
behind them - nails inputs it has seen, falls apart on anything slightly different, like a student who
memorized the practice exam instead of understanding the subject.
📝 **Catastrophic forgetting.** Fine-tuning hard on a narrow task degrades the general abilities the model
used to have - you taught it your support voice so intensely that it got worse at basic reasoning. The cure
is usually a lighter touch, which is exactly what the next section is about.
## Stage 3 - Hosting and serving (the part people forget)
**What it actually is.** A fine-tuned model is *yours* now, which means it has to live somewhere and be
served to your application. With a hosted provider this is mostly handled - you call your tuned model by its
ID. With open-weight models you're running yourself, you own the serving infrastructure: the GPUs, the
scaling, the uptime.
**What it does in real life.** This is the cost that doesn't show up in the proof-of-concept and never goes
away. A base model from a provider is shared infrastructure - you pay per call and someone else keeps it
running. The moment the model is custom-tuned for you, the economics shift toward *you*, directly or folded
into per-call pricing. It's an ongoing line item, not a one-time setup.
**The gotcha.** Teams budget for the training and forget the serving. "We fine-tuned it" is the start of an
operational commitment, not the end of a project. Factor in who keeps the tuned model running, and what it
costs, *before* you commit.
## A lighter way in - LoRA and parameter-efficient tuning
Full fine-tuning - rewriting all of a model's weights - is heavy: expensive to train and a whole model to
store and serve per variant. Most teams don't do that. They use **parameter-efficient fine-tuning**, and the
name you'll hear most is **LoRA**.
📝 **LoRA (Low-Rank Adaptation).** Instead of editing all the model's weights, LoRA freezes the original model
and trains a small set of *new* weights - an "adapter" - that ride alongside it and adjust its behavior. You
train far fewer numbers, so it's cheaper and faster, and the adapter is a small file you can attach to the
base model rather than a whole new copy of it.
**Why this matters for your decision.** LoRA lowers the cost and lock-in of fine-tuning meaningfully. It also
softens catastrophic forgetting, because the original model is left intact underneath the adapter. If you've
clearly decided to fine-tune, parameter-efficient methods like LoRA are usually where to start rather than
full fine-tuning - you get most of the benefit for a fraction of the cost and commitment.
## Stage 4 - Evaluation: how you'd actually know it worked
**What it actually is.** A held-back set of test cases - inputs you did *not* train on, plus a way to judge
the outputs - that tells you whether the tuned model is genuinely better than what you had before. "Before"
means the plain baseline: your best prompt on the base model.
**What it does in real life.** You run the same test inputs through (a) your best-prompted base model and (b)
your fine-tuned model, and compare. Did the format get more consistent? Did the voice match? Did general
ability survive (no catastrophic forgetting)? Without this, you have a vibe, not a result.
**The gotcha.** ⚠️ Evaluate against the *prompted base model*, not a naked, un-prompted one. The fair question
is never "is the tuned model better than nothing?" - it's "is it better than the cheapest thing I could have
done instead?" Skip that comparison and you can convince yourself fine-tuning won when a good prompt would
have tied it for a fraction of the cost.
## Recap
1. Fine-tuning is four stages - **dataset, training, hosting, evaluation** - and they're wildly unequal in
effort.
2. **The dataset is the real cost.** Hundreds to thousands of consistent, high-quality, exact example pairs,
built by someone with domain knowledge.
3. **Garbage data makes a confidently wrong model** that's harder to fix than a base model. Curation is the
job, not the prerequisite.
4. **The training run is mostly a button**; "it finished" doesn't mean "it worked." Watch for overfitting and
catastrophic forgetting.
5. **Hosting/serving is an ongoing cost** people forget to budget for. **LoRA** and parameter-efficient
methods make fine-tuning cheaper and lower lock-in.
6. **Evaluate against your best-prompted base model**, not against nothing.
You now know what the lever actually costs to pull. The final phase turns all of this into a decision you can
make - and defend - in the right order.
---
# Choosing - the No-Nonsense Order
You've got the three levers and you know what fine-tuning really costs. Now the decision. There's a default
order that's right far more often than not, and it falls straight out of one principle: **each step up costs
more and locks you in more, so climb only as high as your problem forces you to.**
This phase gives you that order, a table to decide from in the moment, and the traps that catch nearly
everyone - so you can make the call confidently and explain it to whoever's holding the budget.
## The decision table
> **Start here. Find your situation, then read the section below it.**
| Your situation | Reach for | Why |
|---|---|---|
| "It's not following my instructions / format" | **Prompting** first | You almost certainly haven't exhausted it (§1) |
| "It doesn't know our facts / docs / current data" | **RAG** | Knowledge belongs in context, not in weights (§2) |
| "Facts change often (prices, policies, inventory)" | **RAG** | Retrieval updates instantly; weights go stale (§2) |
| "It knows enough but won't answer in our voice/format reliably, at scale" | **Fine-tuning** | This is behavior, and that's fine-tuning's actual job (§3) |
| "We need shorter prompts / lower latency on a narrow, high-volume task" | **Fine-tuning** (likely LoRA) | Bake the rules into weights instead of paying for them every call (§3) |
| "We want it to *know* our product" | **RAG**, not fine-tuning | Training teaches style, not reliable recall (§1, §2) |
| Not sure yet | **Prompting** | Cheapest place to learn what you actually need (§4) |
## 1. Start with prompting - always
**The move.** Before anything else, push prompting until it genuinely stops improving: clear instructions, a
sharp system prompt, a few well-chosen examples in the context, structured output if you need it. Measure
where it lands.
**Why it's first.** It's free in every way that matters - you edit a string, the behavior changes on the next
call, and you can undo it instantly. No dataset to build, nothing to host, nothing to retrain when the base
model improves. Most "we need to fine-tune" problems dissolve here.
⚠️ **The trap to avoid.** "We tried prompting" almost always means "we tried *a* prompt, once, and it wasn't
perfect." That's not exhausting prompting - that's barely starting. Exhausting it means you've seriously
iterated and hit a real ceiling: instructions ignored, prompts ballooning, consistency stuck at *mostly* when
you need *reliably*. Only a real ceiling justifies climbing higher. Full technique lives in
[Prompt Engineering, Plainly](/guides/prompt-engineering-plainly).
## 2. Add RAG when the problem is knowledge
**The move.** If prompting can't fix it because the model *doesn't know* something - your internal docs,
your current data, anything specific or fresh - add retrieval. Fetch the relevant facts and put them in the
context at request time.
**Why it's second, not last.** RAG costs more than prompting (you're building a retrieval pipeline) but far
less than fine-tuning, and critically, it keeps your facts *editable* - change a document, the next answer is
correct, no retraining. Any problem that's really about knowledge should stop here and never reach
fine-tuning. The full build is in [RAG, Explained](/guides/rag-explained).
💡 **Key point.** If you remember one thing from this whole guide, make it this: **never fine-tune to teach
facts.** Facts change, and weights don't update themselves. The instant your knowledge has a shelf life,
retrieval is the right answer.
## 3. Fine-tune when - and only when - the problem is behavior at scale
**The move.** If prompting is genuinely exhausted *and* the problem is behavior rather than knowledge -
consistent voice, consistent format, consistent style, or a narrow task you run at high volume where you want
the rules baked in - then fine-tune. Reach for parameter-efficient methods like LoRA first (Phase 2).
**The clear "yes" cases.** Fine-tuning earns its cost when:
- You need a **consistent output format or structure** that prompting only achieves *most* of the time, and
"most" isn't good enough.
- You need a **specific voice, tone, or style** as the reliable default across thousands of calls.
- You have a **narrow, repetitive, high-volume task** where baking the behavior into the weights lets you drop
a long instruction prompt - saving tokens and latency on every single call, which adds up at scale.
**The clear "no" cases.** Don't fine-tune when:
- The goal is to make the model **know facts** → that's RAG.
- You **haven't seriously exhausted prompting** → climb back down.
- The behavior you want **changes often** → you'd be retraining constantly.
- The volume is **low** → you'll never recoup the dataset and serving cost; a longer prompt is cheaper.
⚠️ **The staleness trap.** A model you fine-tuned today is built on *today's* base model. Better base models
ship regularly - and when they do, your tuned model doesn't automatically inherit the improvements. To move up,
you re-run your fine-tuning on the new base, which means keeping your dataset and pipeline alive indefinitely.
Prompting and RAG ride the upgrades for free; fine-tuning makes you pay for each one. That ongoing tax is part
of the real cost, and it's exactly why fine-tuning sits *last* in the order.
## 4. Why this order, in one breath
```mermaid
flowchart LR
P["PROMPT change a string instant, free rides model upgrades"] -->|still stuck?| R["RAG build retrieval facts stay editable rides model upgrades"]
R -->|still stuck?| F["FINE-TUNE build a dataset, train, host goes stale on new bases"]
```
*What just happened:* You climb only as far as the problem forces you. Each rung costs more and commits you
more, so the discipline is to *stop at the first rung that actually solves your problem*. Reaching the top is
not a sign of sophistication - reaching exactly the rung you needed is.
## The bottom line, plainly
Fine-tuning is real, useful, and occasionally exactly right - for **behavior** (voice, format, style) on a
**narrow task at scale**, after prompting is genuinely exhausted and RAG doesn't fit because the problem was
never about knowledge. That's a narrower target than the hype suggests. Most teams get where they're going on
prompting and RAG, spend a fraction of the money, and keep the freedom to ride the next better model for free.
If you do climb to fine-tuning, climb on purpose - with a real dataset, a clear baseline, and your eyes open
about the standing cost.
## Recap
1. **Climb the cheapest-first ladder: prompt → RAG → fine-tune.** Each rung costs more and locks you in more;
stop at the first one that solves your problem.
2. **Prompting first, always** - and "we tried it once" is not "we exhausted it."
3. **RAG for knowledge** - and never fine-tune to teach facts, because facts change and weights don't.
4. **Fine-tune only for behavior at scale** - consistent voice/format/style or a narrow high-volume task,
after prompting is truly exhausted; start with LoRA.
5. **Mind the staleness tax** - fine-tuned models don't inherit better base models for free; prompting and RAG
do.
That's the whole decision, plainly put. You can now answer "should we fine-tune our own model?" with a reasoned
order instead of a reflex - and that answer will usually save your team a great deal of time and money.
**Related guides:** [Prompt Engineering, Plainly](/guides/prompt-engineering-plainly) ·
[RAG, Explained](/guides/rag-explained) · [Using an LLM API](/guides/using-an-llm-api)