# Running Models Locally

> What it really means to run an LLM on your own machine - the clear-eyed trade-off against a hosted API, a real Ollama session from download to local API call, and how model size, RAM/VRAM, and quantization decide whether it runs at all.


---

# Running Models Locally

You've used an LLM through a website or an API key, and somewhere along the way a quieter idea took hold: *what if the model just ran on my own machine?* No account, no sending your data to someone else's servers, no meter ticking with every request. It turns out you can - and the experience of pulling a real model down and watching it answer entirely offline is genuinely a little magical the first time.

It's also a trade-off, not a free upgrade. A model running on your laptop is usually weaker than the big hosted ones, and whether it runs *at all* depends on numbers most people have never had explained to them - parameters, RAM, VRAM, quantization. This guide makes those knowable. By the end you'll be able to download a model, talk to it from code, and look at any model on a download page and say "that'll fit my machine" or "that won't" - and know why.

> ⏭️ Never called an LLM from code before? [Using an LLM API](/guides/using-an-llm-api) shows the hosted side first. Running locally is the same idea - text in, text out - with the model living on your hardware instead of someone else's.

## How to read this

- **Just want to decide if local is even worth it?** Read [Phase 1](01-why-run-locally.md) - the clear-eyed trade-off against a hosted API - and stop there if the answer is "not for me yet."
- **Want it to finally make sense?** Read in order. Phase 1 frames the decision, Phase 2 gets a real model running, and Phase 3 explains the hardware reality so you can pick a model that actually fits.

## The phases

1. **[Why (and Why Not) Run Locally](01-why-run-locally.md)** - the clear-eyed trade-off: privacy, zero per-token cost, offline, and control on one side; weaker models, your hardware's limits, and setup effort on the other. When local genuinely makes sense, and when a hosted API is the right call.
2. **[Getting One Running (Ollama)](02-getting-one-running.md)** - the mental model (download an open-weights model, run it locally), then a real `ollama pull` / `ollama run` session, and finally hitting the model's local API endpoint from your own code.
3. **[Hardware, Quantization & Reality](03-hardware-and-quantization.md)** - what actually decides if a model runs: its size in parameters versus your RAM/VRAM, and **quantization** - shrinking the weights to fit, trading a little quality for a lot of memory. CPU versus GPU speed, and how to match a model to your machine.

> This guide gets you running a single model on one machine. Fine-tuning a model on your own data, serving one to a team, and squeezing out maximum speed are each their own topic - deferred to follow-up guides rather than crammed in here.


---

# Why (and Why Not) Run Locally

Before you spend an evening downloading models, it's worth being clear about what you're actually getting. Running a model locally is not "the same thing, but free." It's a genuinely different deal - better in some ways that matter a lot, worse in ways that matter just as much. The people happiest running models locally went in knowing exactly which trade they were making.

Let's lay both sides on the table, then talk about when local is the right call.

## The mental model: where does the model live, and who pays for the compute?

**What it actually is.** There are only two places a large language model can run: on a computer you control, or on a computer someone else controls. That single fact drives almost every trade-off in this phase.

When you call a **hosted API** - OpenAI, Anthropic, and the rest - your text travels over the internet to their data center, runs through a model on their hardware, and the answer travels back. You rent their compute by the token and never touch the model itself. (That whole flow is the subject of [Using an LLM API](/guides/using-an-llm-api).)

When you **run locally**, the model's files sit on your disk, and the math runs on your own CPU or GPU. Nothing leaves your machine. You own the compute, which means you also own its limits.

```mermaid
flowchart LR
  subgraph Hosted["HOSTED API"]
    direction LR
    APP[your app] -->|your text| DC["someone else's data center<br/>+ their big model (you rent it)"]
    DC -->|answer| APP
  end
  subgraph Local["LOCAL - nothing leaves"]
    direction LR
    M["your machine<br/>the model + your app, all in here"]
  end
```

Everything below is a consequence of that picture.

## The case *for* running locally

**Privacy - your data never leaves the machine.** This is the big one. When the model runs locally, the prompt, the documents you feed it, and the answers all stay on your hardware - nothing transmitted to a third party, logged on their servers, or potentially used to train a future model. For sensitive code, private documents, health or legal text, or anything under a confidentiality obligation, this is often the *whole* reason to run locally - sometimes the only acceptable option.

**No per-token cost.** A hosted API bills you for every request, forever. A local model costs you the electricity to run it and nothing else. If you're doing heavy, repetitive work - classifying thousands of records, churning through a big batch overnight, experimenting in a tight loop - the meter that never runs is a real relief.

**Offline.** The model works on a plane, in a basement, behind a corporate firewall, on a flaky connection - anywhere, because there's no network call to fail. Once the files are on disk, the internet is optional.

**Control.** You choose the exact model and version, and it never changes underneath you. A hosted model can be updated, deprecated, rate-limited, or retired on the provider's schedule; your local copy answers the same way today and next year. You can also reach for specialized open-weights models that no big provider offers.

## The case *against* (the plain part)

**The models are usually weaker.** This is the trade you're really making. The largest, sharpest models are enormous - far too big to run on a personal machine - so they're only available as hosted services. The open-weights models you can run at home are smaller, and on hard reasoning, long documents, and tricky instructions, that gap shows. A good local model is genuinely useful; it generally won't match the best hosted model on the hardest tasks. Going in expecting parity is the fastest route to disappointment.

> 📝 **Terminology.** **Open-weights** means the trained model's parameters (its "weights") are published for you to download and run yourself. It's what makes local running possible at all - you can't run a model whose weights nobody released. (It is *not* the same as "open source," and the license attached can still restrict how you use it - more on that in [Phase 3](03-hardware-and-quantization.md).)

**Your hardware is the ceiling.** With an API, the provider's giant machines are the limit. Locally, *your* machine is - its memory decides which models even load, and its CPU or GPU decides how fast they answer. A model that's too big won't run at all, and a model that barely fits may answer slowly enough to test your patience. Phase 3 is entirely about reading those limits before you hit them.

**Setup effort.** A hosted API is a key and an HTTP call. Local means installing a runtime, downloading multi-gigabyte model files, and learning which model suits your hardware. It's very approachable now - Phase 2 walks the whole thing - but it isn't zero, and you maintain it yourself.

## The trade-off at a glance

Here's both sides in one straight table - neither column is the winner; they're different tools.

```text
                      LOCAL                        HOSTED API
  Privacy        data never leaves machine    text sent to provider
  Cost           electricity only             per-token, ongoing
  Offline        yes                          needs a connection
  Control        exact model, never changes   provider's schedule
  Model quality  smaller / weaker             access to the best
  Speed          limited by your hardware     their big machines
  Setup          install + download + tune    an API key
```

## When local genuinely makes sense

Reach for a local model when one of these is true:

- **Privacy is non-negotiable** - sensitive data that mustn't leave your control.
- **You're running a high volume** of cheap, repetitive calls and the per-token bill would sting.
- **You need it offline**, or behind a firewall with no outbound access.
- **You want to learn or tinker** - there's no better way to build a real feel for how these models work than running one yourself.
- **A smaller model is genuinely good enough** for the task - summarizing, drafting, classifying, simple extraction. Plenty of real work doesn't need the very best model.

Lean toward a **hosted API** when you need top-tier quality on hard problems, you don't want to manage infrastructure, or your volume is low enough that the bill is trivial. ⚠️ A common mistake is treating this as all-or-nothing. Plenty of real systems do both - a local model for the bulk, private, or offline work, and a hosted call for the few requests that truly need the strongest model. You're choosing per task, not for life.

## Recap

1. There are two places a model can run: **your machine** or **someone else's** - and that drives every trade-off.
2. **For local:** privacy (data never leaves), no per-token cost, offline, and full control over the model.
3. **Against local:** models are usually weaker, your hardware is the hard ceiling, and there's real setup effort.
4. **Choose local** for privacy, high volume, offline needs, learning, or when a smaller model is good enough; **choose hosted** for top quality on hard tasks with zero infrastructure.
5. It's not all-or-nothing - many systems sensibly use both.

You know the deal you're making. Next, let's actually make it - pull a real model down and talk to it.


---

# Getting One Running (Ollama)

This is the part that feels like a trick the first time it works: you type one command, wait a few minutes, type another, and a real language model is answering you - with the network unplugged, on hardware you can touch. There's no account, no key, no meter.

We'll use **Ollama**, because it removes almost all the friction. It's a small program that handles downloading, storing, and running models - and gives you both a chat prompt and a local API, which is everything you need.

## The mental model: a model is a file you download and run

**What it actually is.** An open-weights model is, at bottom, a big file (or a few files) full of numbers - the trained weights. "Running it locally" means two things working together: a **runtime** (Ollama) that knows how to load those numbers and do the math, and the **model file** itself that the runtime loads. Ollama is the record player; the model is the record.

**Why people get this wrong.** It's easy to picture "installing an AI" as one monolithic thing. Cleaner to keep them separate: install the *runtime* once, then download *models* into it - as many as you like, swapping between them. Pulling a second model doesn't reinstall anything; it just drops another record on the shelf.

```mermaid
flowchart TD
  subgraph Ollama["Ollama (runtime) - loads model files, does the math,<br/>exposes a chat prompt + a local API"]
    A["model A<br/>(weights on disk)"]
    B["model B<br/>(weights on disk)"]
  end
```

> 📝 **Terminology.** When you tell Ollama to **pull** a model, it downloads the weight files to your disk. When you **run** a model, Ollama loads those files into memory and starts answering. Pull once; run as often as you like.

Install Ollama from [ollama.com](https://ollama.com) for your operating system before the steps below - it's a normal installer, and once it's done the `ollama` command is available in your terminal. (Verify with `ollama --version`.)

## `ollama pull` - download a model

Let's bring down a small, capable model. We'll use `llama3.2`, one of Meta's open-weights Llama models, in a size that fits a typical laptop.

```console
$ ollama pull llama3.2
pulling manifest
pulling dde5aa3fc5ff: 100% ▕████████████████▏ 2.0 GB
pulling 966de95ca8a6: 100% ▕████████████████▏ 1.4 KB
pulling fcc5a6bec9da: 100% ▕████████████████▏ 7.7 KB
pulling a70ff7e570d9: 100% ▕████████████████▏ 6.0 KB
pulling 56bb8bd477a5: 100% ▕████████████████▏   96 B
pulling 34bb5ab01051: 100% ▕████████████████▏  561 B
verifying sha256 digest
writing manifest
success
```

*What just happened:* Ollama downloaded the model's files to your disk and verified them. The biggest line - about **2.0 GB** here (an approximate size; it varies by model and version) - is the weights themselves; the small files are metadata. That download happens once. From now on the model lives on your machine, and pulling it again would be instant.

⚠️ **Gotcha.** That number is roughly how much disk *and* memory the model needs. A 2 GB model wants a couple of gigabytes free in RAM to load; bigger models want a lot more. If a model is far larger than your machine's memory, this is where reality bites - covered properly in [Phase 3](03-hardware-and-quantization.md). For now, a model in the low single-digit gigabytes is a safe first choice on most laptops.

## `ollama run` - talk to it in the terminal

```console
$ ollama run llama3.2
>>> In one sentence, what is an open-weights model?
An open-weights model is a machine learning model whose trained
parameters are publicly released, so anyone can download and run
it on their own hardware.

>>> /bye
```

*What just happened:* `ollama run` loaded the model into memory and dropped you into an interactive chat - the `>>>` is its prompt, waiting for you. You typed a question, it answered, all on your machine with no network call. There may be a short pause the first time while the model loads into memory; after that, replies start streaming. Typing `/bye` exits the chat. **That's a real LLM, running entirely on your hardware.**

💡 **Key point.** This terminal chat is the fastest way to *try* a model - pull it, run it, ask it a few real questions, decide if it's good enough for your task before you write a line of code. If it's not, pull a different one and compare. Cheap experiments are one of the quiet joys of running locally.

## Hitting the local API from your code

The terminal chat is for *you*. To build something, you want your program to talk to the model - and Ollama is already serving a local API for exactly that. While Ollama is installed and running, it listens on `http://localhost:11434` (`localhost` means "this same machine," so the request never touches the network).

> ⏭️ If "endpoint," "POST," and "JSON" are fuzzy, [Using an LLM API](/guides/using-an-llm-api) explains the request/response shape first. The pattern here is the same - you're just pointing it at your own machine instead of a provider.

Here's the simplest possible call - a `curl` request from the terminal:

```console
$ curl http://localhost:11434/api/generate -d '{
  "model": "llama3.2",
  "prompt": "Say hello in five words.",
  "stream": false
}'
{"model":"llama3.2","created_at":"2026-06-19T10:12:04Z","response":"Hello there, nice to meet!","done":true}
```

*What just happened:* You sent a POST request to Ollama's local `/api/generate` endpoint with a small JSON body - which **model** to use and the **prompt** to answer. Setting `"stream": false` asks for the whole reply in one response instead of token-by-token, which keeps this example simple to read. Ollama ran the model and sent back JSON; the answer is in the `response` field. No API key, no internet - your machine asked itself a question.

The same call from Python looks like this:

```console
$ python3 - <<'EOF'
import requests

resp = requests.post(
    "http://localhost:11434/api/generate",
    json={
        "model": "llama3.2",
        "prompt": "Say hello in five words.",
        "stream": False,
    },
)
print(resp.json()["response"])
EOF
Hello there, nice to meet!
```

*What just happened:* Exactly the same request, expressed in Python with the `requests` library. You POST the model name and prompt to the local endpoint and read the `response` field out of the returned JSON. From your program's point of view, this is just an HTTP call to `localhost` - anything that can make an HTTP request can use your local model. That's the whole bridge from "a model in my terminal" to "a model in my app."

⚠️ **Gotcha.** If the call fails with "connection refused," the Ollama service isn't running. On most installs it starts in the background automatically; if not, running `ollama serve` (or just opening the Ollama app) starts the listener on port 11434. The API can only answer while that service is up.

## Recap

1. **Ollama is the runtime; models are files** you pull into it - install Ollama once, download as many models as you like.
2. **`ollama pull <model>`** downloads a model's weights to disk (once); the size roughly tells you the disk and memory it needs.
3. **`ollama run <model>`** opens a terminal chat - the fastest way to try a model before coding against it.
4. **`http://localhost:11434/api/generate`** is the local API: POST a JSON body with the model and prompt, read the `response` field - no key, no network.
5. Anything that can make an HTTP request to `localhost` can use your local model.

You can pull a model and talk to it from code. The open question is *which* models your machine can actually handle - and that's pure hardware. Let's make it knowable.


---

# Hardware, Quantization & Reality

This is the phase that turns "I pulled a model and it was painfully slow" or "it just wouldn't load" from a mystery into a prediction you can make *before* you download anything. Whether a model runs, and how fast, comes down to a small number of physical facts about your machine. Once you can read them, model pages stop being a wall of cryptic names and start being a menu you can order from.

There are exactly two questions: **will it fit?** and **will it be fast enough?**

## Will it fit? Parameters versus memory

**What it actually is.** A model's size is measured in **parameters** - the count of numbers (weights) it learned during training. You'll see models labeled `7B`, `13B`, `70B`: that's 7 billion, 13 billion, 70 billion parameters. More parameters generally means a more capable model - and a bigger file that needs more memory to run.

> 📝 **Terminology.** **Parameters** are the learned numbers that *are* the model. "A 7B model" means 7 billion of them - the headline number on almost every open-weights model, and the first thing to look at.

**Why this is the whole game.** To run, every one of those parameters has to be loaded into memory at once, so a model's memory appetite scales directly with its parameter count. That's why a 70B model won't load on a laptop and a 3B model will - it's not subtle, it's arithmetic.

⚠️ **Gotcha - the big one.** A giant model will not fit a small machine, period. If a model needs more memory than you have, it either refuses to load or spills onto disk and crawls so slowly it's unusable. Match the model to the machine *before* you pull it, not after.

**RAM versus VRAM.** There are two kinds of memory that matter, and which one gets used depends on where the model runs:

```text
   ┌──────────────┐         ┌──────────────┐
   │     CPU      │         │     GPU      │
   │  uses RAM    │         │  uses VRAM   │
   │  (system     │         │  (memory on  │
   │   memory)    │         │   the card)  │
   └──────────────┘         └──────────────┘
       8–64 GB                 often 8–24 GB
       typical                 on consumer cards
```

> 📝 **Terminology.** **RAM** is your computer's main system memory. **VRAM** is the separate, faster memory built onto a graphics card. A model running on the GPU must fit in VRAM; a model on the CPU uses ordinary RAM. (Full picture: [CPU, RAM & Storage, Explained](/guides/cpu-ram-and-storage).)

VRAM is usually the tighter constraint, because consumer graphics cards ship with less of it than a machine has system RAM. A model that fits comfortably in 32 GB of RAM might be far too big for an 8 GB graphics card. When checking "will it fit," check against the memory it'll actually use.

## Quantization - shrinking the weights to fit

Here's the lever that makes local models practical - the one concept most worth understanding in this whole guide.

**What it actually is.** Each parameter is a number, and a number can be stored with more or less precision. By default, models are often stored at high precision - many bits per parameter. **Quantization** stores each parameter using *fewer* bits, rounding the numbers to a coarser scale. Fewer bits per parameter means a dramatically smaller file and dramatically less memory needed to run it.

**The trade.** Rounding the weights loses a little accuracy, so a quantized model is slightly less sharp than the full-precision original. But the memory savings are large and the quality loss is usually small - for most everyday use you'd struggle to notice. That lopsided trade - *a little quality for a lot of memory* - is exactly why quantization is the default way people run models locally. It's what lets a model that wouldn't fit your machine suddenly fit.

```mermaid
flowchart LR
  F["FULL PRECISION<br/>big file, needs lots of memory"] -->|round the weights| Q["QUANTIZED<br/>smaller file - fits in less memory,<br/>runs on more machines, slightly less sharp"]
```

**What it looks like in practice.** On model pages and in Ollama's tags you'll see labels like `Q4` or `Q8` - roughly the number of bits per parameter. Lower numbers mean a smaller, lighter model with a bit more quality lost; higher numbers mean larger and closer to the original. A 4-bit quantization (`Q4`) is a common, practical default - small enough to fit modest hardware, good enough for most work. When a model "wouldn't fit," the fix is frequently "use a more quantized version of it."

💡 **Key point.** Two dials decide if a model fits your memory: **how many parameters** (pick a smaller model) and **how aggressively quantized** (pick fewer bits). If a model won't fit, turn one of those dials before giving up.

## Will it be fast enough? CPU versus GPU

Fitting is pass/fail; speed is a spectrum, and it's mostly about *what does the math*.

**What it actually is.** The model's work is an enormous pile of parallel arithmetic. A **GPU** is built for exactly that kind of massively parallel math, so it runs models much faster than a CPU doing the same work. A **CPU** can absolutely run a model - that's what happens on a machine without a capable GPU - just slower, especially as models get bigger.

**What it feels like.** On a GPU with enough VRAM, a reasonably sized model replies briskly, words streaming out at a comfortable reading pace or faster. On a CPU, the same model still works but the words come more slowly, and a large model on CPU can be slow enough to frustrate. (Apple Silicon Macs are a happy middle case - their unified memory lets the GPU portion use system RAM, punching above what you'd expect from the spec sheet.)

⚠️ **Gotcha.** "It loaded but it's crawling" usually means one of two things: the model barely fit and is spilling between memory tiers, or it's running on the CPU when you hoped it'd use the GPU. The fix is the same as for fitting - a smaller or more quantized model that comfortably fits the faster memory.

## One more reality: the license

A practical warning that has nothing to do with hardware.

⚠️ **Gotcha - open-weights licenses vary.** "Open-weights" means you can download and run the model; it does *not* automatically mean you can do anything you want with it. Some models are permissively licensed; others restrict commercial use, large-scale deployment, or specific applications, and some require you to accept terms before downloading. Before you build something real - especially anything commercial - read the license for the specific model. Downloadable weights are not the same as unrestricted use.

## Matching a model to your machine

Put it together as a quick mental checklist, in order:

1. **Start from your memory.** How much RAM? A GPU, and how much VRAM? That's your budget.
2. **Pick a parameter size that fits the budget**, with room to spare - a model should fit *comfortably*, not exactly.
3. **Use a quantized version** (`Q4`-style is a sensible default) to stretch that budget further.
4. **Try it** - `ollama run` it, ask real questions. Good enough and fast enough? Done.
5. **If it doesn't fit or it's too slow**, step down: smaller parameter count, more aggressive quantization, or both.
6. **If quality isn't enough** even at the largest model you can fit, that's the clear signal to use a hosted API for that task - the trade from [Phase 1](01-why-run-locally.md).

> 💡 The numbers in this phase - typical RAM and VRAM amounts, model sizes - are approximate ballparks meant to build intuition, not precise requirements. The real figures shift with each model, version, and quantization, so always check the specific model's page for its actual size before pulling it.

## Recap

1. **Parameters (7B, 13B, 70B…)** measure model size; every parameter must fit in memory to run.
2. A model must fit your **RAM** (if running on CPU) or your **VRAM** (if running on GPU) - VRAM is usually the tighter limit.
3. **Quantization** stores each parameter in fewer bits - much less memory for a little quality loss; it's how big models become runnable, and `Q4`-style builds are a common default.
4. **GPUs run models much faster** than CPUs; "loaded but crawling" usually means it barely fit or is running on the CPU.
5. **Open-weights licenses vary** - downloadable is not the same as unrestricted; read the license before building something real.
6. Match model to machine: start from your memory, pick a size and quantization that fit comfortably, try it, and step down if needed - or reach for a hosted API when local quality isn't enough.

You can now read any model on a download page and predict whether it'll run on your machine, and why. From here, the natural next steps are feeding a local model your own documents and, eventually, fine-tuning one - each its own guide.

**Related guides:** [Using an LLM API](/guides/using-an-llm-api) · [What AI and ML Are](/guides/what-ai-and-ml-are)