Overfitting & Why Test Sets Exist

By now you know training drives the loss down, lap after lap. So here's an uncomfortable question: if lower loss is better, why not keep training until the loss is nearly zero? Wouldn't that be a perfect model?

It would be a model that's perfect on the examples it has already seen - and possibly useless on anything new. That trap has a name, and avoiding it is the reason machine learning has the rituals it does. This phase delivers the difference between a model that memorized and one that actually learned.

Memorizing vs. generalizing

What it actually is. There are two ways for loss to get low. The model can learn the genuine underlying pattern (good), or it can memorize the specific answers to the specific examples it was shown (bad). Both make training loss small. Only the first is worth anything.

📝 Terminology. Generalizing means the model performs well on new inputs it never saw during training - it grasped the pattern. Overfitting is the opposite: the model latched onto the quirks and noise of the training examples themselves, so it aces them and stumbles on everything else.

Why people get this wrong. It's tempting to read "low loss = good model." But low training loss only tells you the model fits the data it studied. A student who memorized last year's exam answer key gets 100% on last year's exam and learned nothing. ⚠️ Overfitting is acing the training data and failing reality - and it's invisible if the only thing you ever measure is performance on the data the model trained on.

   Goal: learn the trend (the line)          Overfitting: trace every point

      price                                     price
        |        .   .                            |        .   .
        |     .  __.--                            |     . _/\_.-\
        |   ._.--                                 |   ._/      \
        |_.--__________ size                      |_/__________ size

   captures the real pattern;               wiggles through every example,
   handles new houses well                  including the noise - fails on new ones

The fix: keep some data hidden

What it actually is. Before training starts, you split your data into separate piles and only let the model learn from one of them. The others are held back specifically to test it on data it has never seen - the only real measure of whether it generalized.

📝 Terminology - the three piles.

Why the split exists. It's the whole point: measuring a model on the data it trained on tells you nothing about the future, the same way grading a student on the answer key they memorized tells you nothing about whether they understand. The held-back data is new to the model, so its score on that data is a fair stand-in for how it'll do on the genuinely new inputs it'll meet in the real world.

A real example.

   Training set:    loss is low      ──►  model fits what it studied
   Test set:        loss is low      ──►  it GENERALIZED   ✓  ship it

   ...vs the bad case...

   Training set:    loss is low      ──►  fits what it studied
   Test set:        loss is high     ──►  it OVERFIT       ✗  memorized, didn't learn

What just happened: The gap between the two scores is the tell. When a model does great on training data but poorly on the held-back test data, that gap is overfitting, made visible. Without a hidden pile of data, you'd never see it coming - you'd ship a model that looked perfect and fell apart in production.

The gotcha. ⚠️ The test set only stays valid if you keep it sealed. The moment you start tweaking your model to score better on the test set, it stops being "data the model never saw" and you're quietly overfitting to it too. That's exactly why the validation set exists as a separate pile: you tune against validation and save the test set for one final, clean look.

A model is only as good as its data

What it actually is. Everything the model knows came from its training examples - and only from them. If those examples are skewed, incomplete, or carry human bias, the model learns the skew right along with the pattern. It has no other source of truth.

What it does in real life. A hiring model trained mostly on one group's résumés learns that group's patterns and quietly disadvantages others. A model trained only on daytime photos struggles at night. The model isn't being malicious or careless - it's faithfully reflecting the data it was handed. "Garbage in, garbage out" is not a warning about the algorithm; it's a fact about where the weights came from.

Why this matters. This is the most important point in the whole guide: a model's fairness and blind spots are inherited from its data, not invented by the math. The data side of that story - where bias creeps in and what to watch for - is covered in ML Basics for Data People.

Why this saves you later. When a model behaves strangely - great in the demo, wrong in the wild; fair for some users, unfair for others - your first question is now the right one: what was it trained on, and what was missing? That instinct will serve you far better than assuming the model is plainly "smart" or "broken."

Recap

1. Low training loss isn't enough - a model can memorize (overfit) instead of learning the real pattern.
2. We split data into training, validation, and test sets so we can measure the model on data it never learned from.
3. A big gap between training performance and test performance is overfitting, made visible.
4. A model inherits the biases and gaps of its data - it has no other source of truth.

So, what is training, really?

Strip away the mystique and here's the whole thing: training is tuning a big pile of numbers until a fixed recipe's guesses match examples we already have answers for - then checking, on data it never saw, that it learned the pattern instead of memorizing the answers.

That's it. Powerful, genuinely useful, occasionally surprising - but not magic. A patient walk downhill, a clean exam at the end, and numbers that are only ever as good as the examples that shaped them.

Watch it animated: overfitting

← Phase 2: Learning by Being Wrong · Guide overview

Related guides: What AI and ML Are · ML Basics for Data People