Why It's Harder Than It Looks

A basic RAG demo takes an afternoon. A RAG system you'd trust with real questions takes a lot longer - and the gap surprises almost everyone. The reason is one sentence worth tattooing on the project:

The model can only be as good as what you retrieve. Garbage in, confident garbage out.

Generation gets the attention because it's the part that talks, but the model is rarely your bottleneck. Retrieval is. If the right chunk never makes it into the prompt, no amount of clever prompting saves you - the model is answering an open-book exam with the wrong page open. This phase is the clear-eyed tour of where that goes wrong, and what to do about it.

The failure-mode cheat-card

Answer looks wrong? Find the symptom, then read the section.

1. Chunking is the quiet make-or-break

Flagged in Phase 2, and it earns its own section because it's the single most underestimated knob.

Why it bites. Your chunks are the only units retrieval can return. Too large, and the embedding is a muddy average of several topics - it matches weakly and pads the prompt with irrelevant text. Too small, and you split a single fact across two chunks, so retrieving one gives the model half a sentence with no context.

   TOO BIG                          TOO SMALL
   ┌──────────────────────┐        ┌──────────┐ ┌──────────┐
   │ setup + testing + CI │        │ "max     │ │ file     │
   │ + deploy, all in one │        │  upload" │ │  size is │
   │ → muddy vector,      │        │          │ │  10 MB"  │
   │   bloated prompt     │        └──────────┘ └──────────┘
   └──────────────────────┘         → the fact is split in two

The calm approach. Chunk along the document's natural seams - headings, sections, logical paragraphs - rather than blindly every N characters. A sane default is to split on structure and allow a small overlap between adjacent chunks so an idea straddling a boundary survives in at least one piece. There's no universal right size; it depends on your docs, which is exactly why §2 (measuring retrieval) matters.

2. Retrieving irrelevant chunks

Why it bites. The vector store always returns your top-k nearest chunks - even when none actually answer the question. "Nearest" is not "relevant." If your docs don't contain the answer, retrieval still hands back the closest chunks, and now the model has confident-looking but off-target context.

A real example.

$ retrieve("what's our refund policy?", k=3)
  0.55  chunk 0012  "Return shipping is paid by the customer unless..."
  0.52  chunk 0077  "Our pricing tiers: Basic, Pro, Enterprise..."
  0.51  chunk 0090  "Support hours are 9–5 ET, Monday–Friday..."

What just happened: Nothing scored high (all near 0.5, versus the 0.91 we saw for a genuine hit in Phase 2). There is no refund-policy chunk, so the store returned the three least irrelevant ones. Feed these to the model and it'll either improvise a refund policy from shipping rules, or - if instructed well - say it doesn't know. The low scores were the warning sign.

The calm approach. Use the similarity scores: set a floor, and treat anything below it as "no good context" rather than forcing an answer. Better retrieval also helps - many production systems combine vector search with old-fashioned keyword search (so exact terms like a product code aren't lost), and some add a reranking step that re-scores candidates more carefully. Reach for those when measurement (below) shows you need them, not on day one.

3. The model ignores or misuses the context

Why it bites. Even with the right chunk in the prompt, the model can lean on its own training-data memory and answer from there, or blend the context with a half-remembered fact and quietly contradict your docs. Giving it context is an invitation, not a handcuff.

The calm approach. A few things move the needle, in rough order of effort:

• Instruct explicitly. Spell out "answer only from the context; if it's not there, say you don't know." The bare version lives in the Phase 2 prompt.
• Demand citations. Asking the model to quote or name the source chunk pushes it to actually use the context - and gives you a way to catch it when it doesn't.
• Mind the ordering. Models can pay less attention to material buried in the middle of a long context. Fewer, better chunks usually beat dumping twenty mediocre ones.

💡 Key point. "Just add more context" is the instinct, and it backfires. A bloated prompt full of marginal chunks dilutes the good one and can increase the chance the model latches onto the wrong detail. Precision beats volume, every time.

4. The index goes stale

Why it bites. Your vectors are a photograph of your docs taken at index time. Edit a doc and the photograph doesn't update itself - the chunk and its vector keep reflecting old text. RAG then confidently retrieves and cites outdated information, arguably worse than not knowing, because it looks sourced.

The calm approach. Treat indexing as an ongoing job, not a one-time setup. Re-index changed documents on a schedule, or trigger a re-index when a source updates. The good news - a genuine advantage over retraining a model - is that refreshing knowledge is cheap: re-embed a handful of changed chunks, not retrain anything. Keeping the index fresh is mostly a matter of remembering to.

5. It still hallucinates when the context is thin

Why it bites. RAG reduces hallucination; it doesn't abolish it. If retrieval comes back thin - nothing relevant, or only a fragment - the model is back in the gap-filling business from Phase 1, now with the extra danger that surrounding context lends its guess a false air of authority.

The calm approach. Accept "I don't know" as a good outcome and design for it: when nothing clears your similarity floor (§2), short-circuit and tell the user you don't have that information rather than asking the model to wing it. A plain "not found" earns far more trust than a confident fabrication.

How you actually keep this straight: evaluate retrieval

Everything above shares one cure: you can't fix what you don't measure, and the thing to measure is retrieval, separately from generation.

Build a small set of real questions paired with the chunk(s) that should be retrieved for each. Run retrieval and check: did the right chunk show up in the top-k? This is the single highest-leverage habit in RAG - it tells you whether a bad answer is a retrieval problem (the right chunk never arrived) or a generation problem (it arrived and the model fumbled it). Without that split, you're tuning blind.

⚠️ Gotcha - don't only eyeball the final answer. A fluent answer can be wrong, and a clunky answer can be perfectly grounded. Judging RAG by how the output reads is how broken retrieval hides in plain sight. Check what was retrieved, not just what was said.

The clear comparison: RAG vs fine-tuning

Sooner or later someone asks: "Why retrieve at all - why not fine-tune the model on our data?" They solve different problems, and conflating them is a common, expensive mistake.

📝 Fine-tuning - continuing to train an existing model on your own examples, adjusting its weights so it absorbs a new behavior, format, or style.

The clear split, both sides:

The rule of thumb: RAG adds knowledge; fine-tuning adds behavior. "The model doesn't know our stuff" is RAG. "The model doesn't answer in our style/format" is fine-tuning. They're not rivals - plenty of serious systems do both: fine-tune for the voice, retrieve for the facts. Deeper on the other half: Fine-Tuning vs Prompting, Plainly.

⏭️ Want to sharpen the instruction half of the augmented prompt - getting the model to obey "answer only from context" reliably? Covered in Prompt Engineering, Plainly.

Recap

1. RAG quality is retrieval quality - the model can't use a chunk you never retrieved.
2. Chunking is the quiet make-or-break: split on natural seams, allow small overlap, and measure.
3. The store always returns something - use similarity scores to reject weak retrievals instead of forcing an answer.
4. Instruct and demand citations so the model uses the context instead of its memory; precision beats volume.
5. Keep the index fresh - stale vectors produce confidently outdated, sourced-looking answers.
6. Evaluate retrieval separately from generation, against known good answers - that's how you know what to fix.
7. RAG adds knowledge; fine-tuning adds behavior. Pick by the problem you actually have, or use both.

← Phase 2: How RAG Works · Guide overview