How RAG Works

Phase 1 landed the idea: fetch the relevant facts, then let the model answer from them. That raises the practical question - how do you find the relevant facts in a pile of thousands of documents, fast, when the user's question won't use the exact same words as your docs?

The pipeline splits cleanly into two stages: a slow indexing stage you do ahead of time, and a fast query stage that runs on every question.

The whole pipeline in one picture

The key insight tying the two halves together: you turn both your documents and the incoming question into the same kind of object - a vector - so you measure how related they are by math, not keyword matching. Walk through each step.

Step 1 - Chunk your documents

What it actually is. Chunking is splitting each document into smaller pieces - a few paragraphs each, roughly. You don't embed a whole 40-page PDF as one blob; you break it into bite-sized chunks.

Why bother. You want retrieval to return the relevant paragraph, not an entire manual - context windows are finite and you want to spend them on signal. A vector representing one focused idea is also far more useful for matching than one averaging a whole document into a single point.

A real example. A 12-page onboarding doc becomes a list of chunks:

chunk 0001  "## Setting up your laptop - Request access to the VPN via..."
chunk 0002  "## Cloning the monorepo - Use SSH, not HTTPS. Run git clone..."
chunk 0003  "## Running the test suite - `make test` runs unit tests; CI also..."

What just happened: One document became several independently-retrievable pieces, each focused on a single topic. When someone later asks "how do I run the tests?", you want chunk 0003 alone to surface - not the whole onboarding doc.

⚠️ Gotcha - chunking is a real decision, not a formality. Chunk too big and each piece is muddy and dilutes the prompt; chunk too small and you slice a single idea in half so neither piece makes sense alone. More on why this is the quiet make-or-break of RAG in Phase 3.

Step 2 - Embed each chunk

What it actually is. An embedding is a list of numbers (a vector) that represents the meaning of a piece of text. Run a chunk through an embedding model and you get back something like [0.021, -0.44, 0.18, ...] - typically hundreds of numbers. The crucial property: texts that mean similar things land at nearby points, even sharing no words. "How do I run the tests?" and "executing the test suite" end up close together.

📝 Embedding - a numeric vector that captures the meaning of text, produced by an embedding model. Closeness between two vectors ≈ closeness in meaning. The full story lives in Embeddings and Vector Search - for RAG you only need the property above.

What it does in real life. Embed every chunk and store the resulting vectors in a vector store (also called a vector database or index) - a system built to hold millions of vectors and answer "which stored vectors are nearest to this one?" quickly.

A real example.

chunk 0003  "Running the test suite - make test runs unit tests..."
            │
            ▼  embedding model
   vector   [ 0.08, -0.21, 0.55, ... ]   ──►  stored in vector index

What just happened: That chunk is now a point in "meaning space," parked in the index next to other chunks about testing, building, and CI. Steps 1 and 2 are the slow part - redo them only when your docs change.

Step 3 - At query time, embed the question

What it actually is. When a question arrives, run it through the same embedding model used on the chunks. Same model matters - both have to live in the same meaning space for the comparison to mean anything.

A real example.

question  "where do I find the command to run tests?"
          │
          ▼  same embedding model
  vector  [ 0.07, -0.19, 0.58, ... ]   ←  notice: close to chunk 0003's vector

What just happened: The question is now a point in the same space as your chunks - and it landed near chunk 0003, even though the question said "command to run tests" and the chunk said "make test runs unit tests." No shared keywords, but similar meaning, so similar vectors. This is why embeddings beat keyword search for this job.

Step 4 - Retrieve the top-k chunks

What it actually is. Ask the vector store for the top-k chunks whose vectors are nearest to the question's vector - the k most relevant pieces. k is just how many you pull back; 3 to 8 is a common starting range.

A real example.

$ retrieve(question_vector, k=3)
  0.91  chunk 0003  "Running the test suite - make test runs unit tests..."
  0.74  chunk 0002  "Cloning the monorepo - use SSH, not HTTPS..."
  0.69  chunk 0048  "CI pipeline - every PR triggers the full test run..."

What just happened: The store ranked every chunk by closeness to the question and handed back the top 3, with a similarity score each. Chunk 0003 is the clear winner; the other two are loosely related. These are your "open-book pages" - the facts about to go to the model. (Notice the system retrieves whatever ranks highest, relevant or not - keep that in mind for Phase 3.)

Step 5 - Build the augmented prompt

What it actually is. Assemble a single prompt that stitches the retrieved chunks together as context, adds a clear instruction to answer from that context, then includes the user's actual question. This is the "augmented" in Retrieval-Augmented Generation.

A real example - an annotated augmented prompt.

┌─ SYSTEM / INSTRUCTION ──────────────────────────────────────┐
│ Answer the question using ONLY the context below.           │  ← tells the model
│ If the context doesn't contain the answer, say you don't    │     to stay grounded,
│ know. Cite the source chunk you used.                       │     and to admit gaps
├─ CONTEXT (retrieved chunks) ────────────────────────────────┤
│ [chunk 0003] Running the test suite - `make test` runs the  │  ← the facts you
│   unit tests. CI also runs them on every PR.                │     fetched in step 4,
│ [chunk 0048] CI pipeline - every PR triggers the full test  │     pasted in verbatim
│   run before merge.                                         │
├─ USER QUESTION ─────────────────────────────────────────────┤
│ Where do I find the command to run tests?                   │  ← what the user
└─────────────────────────────────────────────────────────────┘     actually asked

What just happened: You built the open-book exam paper. The instruction sets the rules ("use only this, admit if it's not here, cite it"), the context is the textbook pages you looked up, and the question is the question. The model never had to recall a thing - everything it needs is right there.

💡 Key point. That instruction line - "answer only from the context, and say you don't know if it isn't there" - is doing real work. It's your main lever for pushing the model away from its generic memory. It isn't a guarantee (more in Phase 3), but it changes behavior meaningfully.

Step 6 - Generate

What it actually is. Send that assembled prompt to the LLM through its API and get the answer back. If you're new to calling a model programmatically, that mechanical step - building the request, sending it, handling the response - is covered in Using an LLM API.

A real example.

LLM:  Run `make test` - that runs the unit tests locally. CI also
      runs the full suite on every PR before merge.  [source: chunk 0003]

What just happened: The model read the context, answered from it, and pointed at the chunk it used. That's a grounded, verifiable answer - a human can click through to chunk 0003 and confirm it. That citability is one of RAG's biggest practical wins, and it falls out of the pipeline naturally because you already know which chunks you handed over.

Recap

1. Indexing (ahead of time): chunk your docs → embed each chunk → store the vectors in a vector store.
2. Querying (every question): embed the question with the same model → retrieve the top-k nearest chunks → build an augmented prompt → generate.
3. Both documents and the question become vectors, so you match on meaning, not keywords.
4. The augmented prompt = instruction ("answer only from this, cite it") + retrieved context + the user's question.
5. Because you know which chunks you sent, you can cite sources - grounded answers a human can verify.

You now have a working mental model of the machine. The catch: this clean pipeline is deceptively easy to build badly. The next phase is the plain-spoken one - why real RAG is mostly a retrieval problem, and where it bites.

← Phase 1: The Problem RAG Solves · Guide overview · Phase 3: Why It's Harder Than It Looks →