# RAG (Retrieval-Augmented Generation), Explained

> What RAG actually is - retrieving the right facts from your own data first, then asking the model to answer using them - why an LLM needs it, how the pipeline works end to end, and why good RAG is mostly good retrieval.


---

# RAG (Retrieval-Augmented Generation), Explained

You've watched an LLM confidently invent a function that doesn't exist, cite a policy your company never wrote, or shrug at a question it should have known the answer to. The model isn't broken - it just doesn't *have* your data. It only knows what it absorbed during training: a frozen, generic snapshot of the public internet, with nothing about your codebase, your docs, or last Tuesday's incident.

RAG is the standard fix. The idea is calmer than the acronym suggests: before you ask the model anything, you go and *fetch the relevant facts* from your own data, then hand them to the model and say "answer using these." This guide builds that idea up properly - what problem it solves, the exact pipeline that makes it work, and the real reasons it's harder to do well than the diagrams suggest.

## How to read this

- **Just need the gist of what RAG is?** Read [Phase 1: The Problem RAG Solves](01-the-problem-rag-solves.md) - it gives you the whole mental model in one sitting.
- **Want it to actually make sense?** Read in order. Phase 1 is the *why*, Phase 2 is the *how*, and Phase 3 is the *why it's harder than it looks* - the part that separates a demo from something you'd trust in production.

## The phases

1. **[The Problem RAG Solves](01-the-problem-rag-solves.md)** - why an LLM alone makes things up about your data, and the open-book-exam mental model that fixes it.
2. **[How RAG Works](02-how-rag-works.md)** - the pipeline: chunk your docs, embed them into a vector store, retrieve the most relevant chunks at query time, stuff them into the prompt, and generate.
3. **[Why It's Harder Than It Looks](03-why-its-harder-than-it-looks.md)** - RAG quality is retrieval quality. Bad chunking, ignored context, stale indexes, thin retrieval, and the clear line between RAG and fine-tuning.

> Deliberately deferred: the deep mechanics of *how* embeddings and vector search work live in their own guide, [Embeddings and Vector Search](/guides/embeddings-and-vector-search). This guide uses them as a building block and links there when you want to go deeper.


---

# The Problem RAG Solves

Picture asking a brilliant new hire a question about your company on their first morning. They're sharp, well-read, articulate - and they know *nothing specific* about your systems. If you press them for an answer about your internal billing flow, the worst thing they can do is *guess fluently*: produce a confident, well-phrased answer that's flatly wrong.

That's an LLM on its own. Before we can appreciate what RAG does, we have to be clear-eyed about what the model actually knows - and what it doesn't.

## What an LLM actually knows

**What it actually is.** A large language model is a giant pattern-predictor trained once, on a fixed pile of text, up to a fixed cutoff date. Everything it "knows" is baked into its weights during that training run. It has no live connection to your files, your database, or today's events. When you chat with it, it isn't *looking anything up* - it's predicting the most plausible next words based on patterns it absorbed.

**Why people get this wrong.** It *feels* like the model is reasoning from facts, because the prose is so fluent. So when it answers a question about your internal API, it's easy to assume it went and checked something. It didn't. It produced text that *sounds* like a correct answer.

This leaves you with three concrete gaps:

- **It's stale.** Training has a cutoff. Anything that happened, changed, or shipped after that date is invisible to the model.
- **It's generic.** It learned from public text. Your private repo, your wiki, your ticket history, your contracts - none of that was in the training data.
- **It fills gaps by inventing.** When the model doesn't know, it rarely says so. It produces the most statistically plausible-sounding answer, which can be entirely fabricated. This is what people mean by **hallucination**.

> 📝 **Hallucination** - when a model generates confident, fluent output that is factually wrong or made up. It isn't lying; it has no concept of truth, only of plausibility. It's filling a gap the only way it can.

**A real example.** Here's the failure mode, with no retrieval in play:

```text
You:   What's the maximum file size our /upload endpoint accepts?

LLM:   The /upload endpoint accepts files up to 25 MB by default,
       configurable via the MAX_UPLOAD_SIZE environment variable.
```

*What just happened:* That answer is plausible, specific, and formatted like documentation - and the model has never seen your codebase. The "25 MB" and the env-var name are pattern-matched from similar public projects, not read from your config. It might be right by luck. You have no way to tell, and neither does the model. This is exactly the kind of answer that ruins an afternoon when someone trusts it.

## The fix: don't make the model recall - let it read

Here's the shift in thinking that RAG is built on.

The model is great at *reading and reasoning over text you give it*. It's unreliable at *recalling specific facts from memory*. So stop relying on its memory. Instead, go fetch the relevant facts yourself, paste them into the conversation, and ask the model to answer **using only what you provided**.

💡 **Key point - the open-book exam.** A closed-book exam tests memory: you either memorized the fact or you didn't, and under pressure you might bluff. An open-book exam is different - you're handed the textbook, you look up the relevant page, and you answer from it. RAG turns every question into an open-book exam for the model. You do the "look up the relevant page" part; the model does the "read it and answer" part, which is what it's actually good at.

That's the whole idea in the name:

- **Retrieval** - go find the relevant facts from your data.
- **Augmented** - add those facts to the prompt.
- **Generation** - let the model generate its answer from them.

**A real example.** Same question, now open-book:

```text
[Retrieved from your repo - config/upload.yaml]
  max_file_size_mb: 10
  allowed_types: [pdf, png, jpg]

You:   Using the context above, what's the maximum file size
       our /upload endpoint accepts?

LLM:   Your /upload endpoint accepts files up to 10 MB, and only
       PDF, PNG, and JPG types.
```

*What just happened:* The model didn't recall anything. You retrieved the actual config from your repo, placed it in the prompt as context, and the model read it and answered from it. The answer is now grounded in *your* data - and because the source is right there, you can verify it. The fluent-bluffing failure mode is largely closed off, because the model is reading rather than guessing.

## Why this is the standard approach

You might wonder why we don't just *retrain* the model on your data. We'll cover that trade-off plainly in [Phase 3](03-why-its-harder-than-it-looks.md), but the short version: retrieval is cheap, fast to update, and lets you point to your sources. When a doc changes, you re-index a file - you don't retrain a model. And because the facts are sitting in the prompt, you can show the user *where the answer came from*, which is the difference between "trust me" and "here's the source."

## Recap

1. An LLM only knows its **training data** - which is stale, generic, and blind to your private docs.
2. When it doesn't know, it doesn't stop - it **hallucinates** a plausible-sounding answer.
3. The model is good at **reading and reasoning over text you give it**, weak at recalling specific facts.
4. **RAG** = retrieve the relevant facts first, add them to the prompt, then let the model generate from them.
5. The mental model is an **open-book exam**: you look up the page, the model reads it and answers.

Now you know *why* RAG exists. Next, the actual machinery - how you turn a pile of documents into something you can retrieve from in milliseconds.


---

# How RAG Works

[Phase 1](01-the-problem-rag-solves.md) landed the idea: fetch the relevant facts, then let the model answer from them. That raises the practical question - how do you *find* the relevant facts in a pile of thousands of documents, fast, when the user's question won't use the exact same words as your docs?

The pipeline splits cleanly into two stages: a slow **indexing** stage you do ahead of time, and a fast **query** stage that runs on every question.

## The whole pipeline in one picture

```mermaid
flowchart LR
  subgraph Indexing["INDEXING - ahead of time, redone when docs change"]
    direction LR
    D[your docs] --> CH[chunk] --> E[embed each chunk] --> VS[(vector store)]
  end
  subgraph Query["QUERY - on every question"]
    direction LR
    Q[question] --> QE[embed question] --> S[search nearest] --> K[top-k chunks] --> PR[augmented prompt] --> L[LLM generates] --> A[grounded answer]
  end
  VS -.->|searched against| S
```

The key insight tying the two halves together: you turn *both* your documents and the incoming question into the same kind of object - a **vector** - so you measure how related they are by math, not keyword matching. Walk through each step.

## Step 1 - Chunk your documents

**What it actually is.** Chunking is splitting each document into smaller pieces - a few paragraphs each, roughly. You don't embed a whole 40-page PDF as one blob; you break it into bite-sized chunks.

**Why bother.** You want retrieval to return the *relevant paragraph*, not an entire manual - context windows are finite and you want to spend them on signal. A vector representing one focused idea is also far more useful for matching than one averaging a whole document into a single point.

**A real example.** A 12-page onboarding doc becomes a list of chunks:

```text
chunk 0001  "## Setting up your laptop - Request access to the VPN via..."
chunk 0002  "## Cloning the monorepo - Use SSH, not HTTPS. Run git clone..."
chunk 0003  "## Running the test suite - `make test` runs unit tests; CI also..."
```

*What just happened:* One document became several independently-retrievable pieces, each focused on a single topic. When someone later asks "how do I run the tests?", you want chunk 0003 alone to surface - not the whole onboarding doc.

> ⚠️ **Gotcha - chunking is a real decision, not a formality.** Chunk too big and each piece is muddy and dilutes the prompt; chunk too small and you slice a single idea in half so neither piece makes sense alone. More on why this is the quiet make-or-break of RAG in [Phase 3](03-why-its-harder-than-it-looks.md).

## Step 2 - Embed each chunk

**What it actually is.** An **embedding** is a list of numbers (a vector) that represents the *meaning* of a piece of text. Run a chunk through an embedding model and you get back something like `[0.021, -0.44, 0.18, ...]` - typically hundreds of numbers. The crucial property: texts that mean similar things land at nearby points, even sharing no words. "How do I run the tests?" and "executing the test suite" end up close together.

> 📝 **Embedding** - a numeric vector that captures the meaning of text, produced by an embedding model. Closeness between two vectors ≈ closeness in meaning. The full story lives in [Embeddings and Vector Search](/guides/embeddings-and-vector-search) - for RAG you only need the property above.

**What it does in real life.** Embed every chunk and store the resulting vectors in a **vector store** (also called a vector database or index) - a system built to hold millions of vectors and answer "which stored vectors are nearest to *this* one?" quickly.

**A real example.**

```text
chunk 0003  "Running the test suite - make test runs unit tests..."
            │
            ▼  embedding model
   vector   [ 0.08, -0.21, 0.55, ... ]   ──►  stored in vector index
```

*What just happened:* That chunk is now a point in "meaning space," parked in the index next to other chunks about testing, building, and CI. Steps 1 and 2 are the slow part - redo them only when your docs change.

## Step 3 - At query time, embed the question

**What it actually is.** When a question arrives, run it through the *same* embedding model used on the chunks. Same model matters - both have to live in the same meaning space for the comparison to mean anything.

**A real example.**

```text
question  "where do I find the command to run tests?"
          │
          ▼  same embedding model
  vector  [ 0.07, -0.19, 0.58, ... ]   ←  notice: close to chunk 0003's vector
```

*What just happened:* The question is now a point in the same space as your chunks - and it landed near chunk 0003, even though the question said "command to run tests" and the chunk said "make test runs unit tests." No shared keywords, but similar *meaning*, so similar vectors. This is why embeddings beat keyword search for this job.

## Step 4 - Retrieve the top-k chunks

**What it actually is.** Ask the vector store for the **top-k** chunks whose vectors are nearest to the question's vector - the *k* most relevant pieces. `k` is just how many you pull back; 3 to 8 is a common starting range.

**A real example.**

```text
$ retrieve(question_vector, k=3)
  0.91  chunk 0003  "Running the test suite - make test runs unit tests..."
  0.74  chunk 0002  "Cloning the monorepo - use SSH, not HTTPS..."
  0.69  chunk 0048  "CI pipeline - every PR triggers the full test run..."
```

*What just happened:* The store ranked every chunk by closeness to the question and handed back the top 3, with a similarity score each. Chunk 0003 is the clear winner; the other two are loosely related. These are your "open-book pages" - the facts about to go to the model. (Notice the system retrieves whatever ranks highest, relevant or not - keep that in mind for Phase 3.)

## Step 5 - Build the augmented prompt

**What it actually is.** Assemble a single prompt that stitches the retrieved chunks together as *context*, adds a clear instruction to answer **from that context**, then includes the user's actual question. This is the "augmented" in Retrieval-Augmented Generation.

**A real example - an annotated augmented prompt.**

```text
┌─ SYSTEM / INSTRUCTION ──────────────────────────────────────┐
│ Answer the question using ONLY the context below.           │  ← tells the model
│ If the context doesn't contain the answer, say you don't    │     to stay grounded,
│ know. Cite the source chunk you used.                       │     and to admit gaps
├─ CONTEXT (retrieved chunks) ────────────────────────────────┤
│ [chunk 0003] Running the test suite - `make test` runs the  │  ← the facts you
│   unit tests. CI also runs them on every PR.                │     fetched in step 4,
│ [chunk 0048] CI pipeline - every PR triggers the full test  │     pasted in verbatim
│   run before merge.                                         │
├─ USER QUESTION ─────────────────────────────────────────────┤
│ Where do I find the command to run tests?                   │  ← what the user
└─────────────────────────────────────────────────────────────┘     actually asked
```

*What just happened:* You built the open-book exam paper. The instruction sets the rules ("use only this, admit if it's not here, cite it"), the context is the textbook pages you looked up, and the question is the question. The model never had to recall a thing - everything it needs is right there.

> 💡 **Key point.** That instruction line - "answer only from the context, and say you don't know if it isn't there" - is doing real work. It's your main lever for pushing the model away from its generic memory. It isn't a guarantee (more in Phase 3), but it changes behavior meaningfully.

## Step 6 - Generate

**What it actually is.** Send that assembled prompt to the LLM through its API and get the answer back. If you're new to calling a model programmatically, that mechanical step - building the request, sending it, handling the response - is covered in [Using an LLM API](/guides/using-an-llm-api).

**A real example.**

```text
LLM:  Run `make test` - that runs the unit tests locally. CI also
      runs the full suite on every PR before merge.  [source: chunk 0003]
```

*What just happened:* The model read the context, answered from it, and pointed at the chunk it used. That's a grounded, *verifiable* answer - a human can click through to chunk 0003 and confirm it. That citability is one of RAG's biggest practical wins, and it falls out of the pipeline naturally because you already know which chunks you handed over.

## Recap

1. **Indexing (ahead of time):** chunk your docs → embed each chunk → store the vectors in a vector store.
2. **Querying (every question):** embed the question with the *same* model → retrieve the top-k nearest chunks → build an augmented prompt → generate.
3. Both documents and the question become **vectors**, so you match on *meaning*, not keywords.
4. The **augmented prompt** = instruction ("answer only from this, cite it") + retrieved context + the user's question.
5. Because you know which chunks you sent, you can **cite sources** - grounded answers a human can verify.

You now have a working mental model of the machine. The catch: this clean pipeline is deceptively easy to build badly. The next phase is the plain-spoken one - why real RAG is mostly a retrieval problem, and where it bites.


---

# Why It's Harder Than It Looks

A basic RAG demo takes an afternoon. A RAG system you'd trust with real questions takes a lot longer - and the gap surprises almost everyone. The reason is one sentence worth tattooing on the project:

> **The model can only be as good as what you retrieve. Garbage in, confident garbage out.**

Generation gets the attention because it's the part that talks, but the model is rarely your bottleneck. **Retrieval is.** If the right chunk never makes it into the prompt, no amount of clever prompting saves you - the model is answering an open-book exam with the wrong page open. This phase is the clear-eyed tour of where that goes wrong, and what to do about it.

## The failure-mode cheat-card

> **Answer looks wrong? Find the symptom, then read the section.**

| Symptom | What's probably happening | Section |
|---|---|---|
| Answers are vague or miss the obvious detail | Chunks too big/small - the right idea got diluted or sliced | §1 |
| The right info exists but never shows up in answers | Retrieval is pulling the wrong chunks | §2 |
| Model ignores the context and answers from memory | Weak instruction, or context buried/contradicted | §3 |
| Answers are confidently outdated | Stale index - docs changed, vectors didn't | §4 |
| Still makes things up despite retrieval | Context was thin or absent; model filled the gap | §5 |

## 1. Chunking is the quiet make-or-break

Flagged in [Phase 2](02-how-rag-works.md), and it earns its own section because it's the single most underestimated knob.

**Why it bites.** Your chunks are the *only* units retrieval can return. Too large, and the embedding is a muddy average of several topics - it matches weakly and pads the prompt with irrelevant text. Too small, and you split a single fact across two chunks, so retrieving one gives the model half a sentence with no context.

```text
   TOO BIG                          TOO SMALL
   ┌──────────────────────┐        ┌──────────┐ ┌──────────┐
   │ setup + testing + CI │        │ "max     │ │ file     │
   │ + deploy, all in one │        │  upload" │ │  size is │
   │ → muddy vector,      │        │          │ │  10 MB"  │
   │   bloated prompt     │        └──────────┘ └──────────┘
   └──────────────────────┘         → the fact is split in two
```

**The calm approach.** Chunk along the document's natural seams - headings, sections, logical paragraphs - rather than blindly every N characters. A sane default is to split on structure and allow a small *overlap* between adjacent chunks so an idea straddling a boundary survives in at least one piece. There's no universal right size; it depends on your docs, which is exactly why §2 (measuring retrieval) matters.

## 2. Retrieving irrelevant chunks

**Why it bites.** The vector store *always* returns your top-k nearest chunks - even when none actually answer the question. "Nearest" is not "relevant." If your docs don't contain the answer, retrieval still hands back the *closest* chunks, and now the model has confident-looking but off-target context.

**A real example.**

```text
$ retrieve("what's our refund policy?", k=3)
  0.55  chunk 0012  "Return shipping is paid by the customer unless..."
  0.52  chunk 0077  "Our pricing tiers: Basic, Pro, Enterprise..."
  0.51  chunk 0090  "Support hours are 9–5 ET, Monday–Friday..."
```

*What just happened:* Nothing scored high (all near 0.5, versus the 0.91 we saw for a genuine hit in Phase 2). There is no refund-policy chunk, so the store returned the three *least irrelevant* ones. Feed these to the model and it'll either improvise a refund policy from shipping rules, or - if instructed well - say it doesn't know. The low scores were the warning sign.

**The calm approach.** Use the similarity scores: set a floor, and treat anything below it as "no good context" rather than forcing an answer. Better retrieval also helps - many production systems combine vector search with old-fashioned keyword search (so exact terms like a product code aren't lost), and some add a *reranking* step that re-scores candidates more carefully. Reach for those when measurement (below) shows you need them, not on day one.

## 3. The model ignores or misuses the context

**Why it bites.** Even with the right chunk in the prompt, the model can lean on its own training-data memory and answer from there, or blend the context with a half-remembered fact and quietly contradict your docs. Giving it context is an invitation, not a handcuff.

**The calm approach.** A few things move the needle, in rough order of effort:

- **Instruct explicitly.** Spell out "answer *only* from the context; if it's not there, say you don't know." The bare version lives in the Phase 2 prompt.
- **Demand citations.** Asking the model to quote or name the source chunk pushes it to actually use the context - and gives you a way to catch it when it doesn't.
- **Mind the ordering.** Models can pay less attention to material buried in the middle of a long context. Fewer, better chunks usually beat dumping twenty mediocre ones.

> 💡 **Key point.** "Just add more context" is the instinct, and it backfires. A bloated prompt full of marginal chunks dilutes the good one and can *increase* the chance the model latches onto the wrong detail. Precision beats volume, every time.

## 4. The index goes stale

**Why it bites.** Your vectors are a *photograph* of your docs taken at index time. Edit a doc and the photograph doesn't update itself - the chunk and its vector keep reflecting old text. RAG then confidently retrieves and cites outdated information, arguably worse than not knowing, because it *looks* sourced.

**The calm approach.** Treat indexing as an ongoing job, not a one-time setup. Re-index changed documents on a schedule, or trigger a re-index when a source updates. The good news - a genuine advantage over retraining a model - is that refreshing knowledge is cheap: re-embed a handful of changed chunks, not retrain anything. Keeping the index fresh is mostly a matter of remembering to.

## 5. It still hallucinates when the context is thin

**Why it bites.** RAG reduces hallucination; it doesn't abolish it. If retrieval comes back thin - nothing relevant, or only a fragment - the model is back in the gap-filling business from [Phase 1](01-the-problem-rag-solves.md), now with the extra danger that surrounding context lends its guess a false air of authority.

**The calm approach.** Accept "I don't know" as a *good* outcome and design for it: when nothing clears your similarity floor (§2), short-circuit and tell the user you don't have that information rather than asking the model to wing it. A plain "not found" earns far more trust than a confident fabrication.

## How you actually keep this straight: evaluate retrieval

Everything above shares one cure: you can't fix what you don't measure, and the thing to measure is **retrieval**, separately from generation.

Build a small set of real questions paired with the chunk(s) that *should* be retrieved for each. Run retrieval and check: did the right chunk show up in the top-k? This is the single highest-leverage habit in RAG - it tells you whether a bad answer is a *retrieval* problem (the right chunk never arrived) or a *generation* problem (it arrived and the model fumbled it). Without that split, you're tuning blind.

> ⚠️ **Gotcha - don't only eyeball the final answer.** A fluent answer can be wrong, and a clunky answer can be perfectly grounded. Judging RAG by how the output *reads* is how broken retrieval hides in plain sight. Check what was retrieved, not just what was said.

## The clear comparison: RAG vs fine-tuning

Sooner or later someone asks: "Why retrieve at all - why not *fine-tune* the model on our data?" They solve different problems, and conflating them is a common, expensive mistake.

> 📝 **Fine-tuning** - continuing to train an existing model on your own examples, adjusting its weights so it absorbs a new *behavior, format, or style*.

The clear split, both sides:

| | **RAG** | **Fine-tuning** |
|---|---|---|
| What it changes | What the model *knows* (facts in the prompt) | How the model *behaves* (style, format, tone) |
| Best for | Injecting your facts, docs, knowledge | Teaching a consistent voice, structure, or task pattern |
| Updating | Re-index changed docs - cheap, fast | Retrain on new examples - slower, costlier |
| Can cite sources? | Yes - the facts are right there | No - knowledge is baked into weights |
| Risk | Bad retrieval → bad answer | Stale/expensive to refresh; can still hallucinate facts |

The rule of thumb: **RAG adds knowledge; fine-tuning adds behavior.** "The model doesn't know our stuff" is RAG. "The model doesn't *answer in our style/format*" is fine-tuning. They're not rivals - plenty of serious systems do both: fine-tune for the voice, retrieve for the facts. Deeper on the other half: [Fine-Tuning vs Prompting, Plainly](/guides/fine-tuning-vs-prompting).

> ⏭️ Want to sharpen the *instruction* half of the augmented prompt - getting the model to obey "answer only from context" reliably? Covered in [Prompt Engineering, Plainly](/guides/prompt-engineering-plainly).

## Recap

1. **RAG quality is retrieval quality** - the model can't use a chunk you never retrieved.
2. **Chunking** is the quiet make-or-break: split on natural seams, allow small overlap, and measure.
3. The store always returns *something* - use **similarity scores** to reject weak retrievals instead of forcing an answer.
4. **Instruct and demand citations** so the model uses the context instead of its memory; precision beats volume.
5. **Keep the index fresh** - stale vectors produce confidently outdated, sourced-looking answers.
6. **Evaluate retrieval separately** from generation, against known good answers - that's how you know what to fix.
7. **RAG adds knowledge; fine-tuning adds behavior.** Pick by the problem you actually have, or use both.