# ML Basics for Data People
> What machine learning actually is from a data perspective - learning patterns from history instead of hand-writing rules, the workflow from features to evaluation, and why ML lives or dies on your data.
---
# ML Basics for Data People
You already work with data. You write SQL, build dashboards, and can smell a bad join from across the room. Now machine learning keeps coming up - in standups, in job posts, in "can't we just throw a model at it?" meetings - and it feels like a different world with its own priesthood and vocabulary.
Here's the reassuring truth: ML is not a different world. It's a different *technique* applied to the same raw material you already handle every day - data. The hard part of ML is almost never the math or the model; it's the data work you already understand. This guide gives you enough of a mental model to follow the conversation, ask the right questions, and recognize where your existing skills are exactly what a project needs.
## How to read this
- **Want the one-paragraph version?** ML learns patterns from historical data to make predictions on new data, instead of you hand-writing the rules. Everything else is detail. Read [Phase 1](01-what-ml-actually-is.md) and you'll have the core idea.
- **Want it to finally make sense?** Read in order. Each phase builds on the last: what ML *is*, how a project actually flows, and where you - the data person - fit and why you matter more than you think.
## The phases
1. **[What ML Actually Is (for Data People)](01-what-ml-actually-is.md)** - learning patterns from examples instead of writing rules by hand; the difference between supervised and unsupervised, grounded in a churn example.
2. **[The Workflow](02-the-workflow.md)** - features, splitting into train and test (and *why*), training, and evaluating - including why "99% accurate" can still be a useless model.
3. **[Where Data People Fit](03-where-data-people-fit.md)** - the unglamorous truth: clean inputs, good features, leak-free splits, reliable pipelines. The model is the easy part.
> This guide stops at the *basics* on purpose. Deep learning, neural networks, and large language models (the "AI" everyone's talking about) are their own territory - we'll point you toward a future **ai-ml** category for that, rather than cram it in here. The foundations below are what make that material make sense later.
> Related reading: [What Is Data Engineering](/guides/what-is-data-engineering) and [Data Quality and Observability](/guides/data-quality-and-observability) - the disciplines that feed ML its lifeblood.
---
# What ML Actually Is (for Data People)
Before any tools or algorithms, here's the one idea the whole field rests on. Once you have it, ML stops being intimidating and starts being a set of choices you can reason about.
## The shift: from writing rules to learning them
Think about how you'd normally answer a question like "which customers are about to cancel?" You'd sit down and write rules - in SQL, in a spreadsheet, in your head:
```text
THE OLD WAY: you write the rules
IF days_since_last_login > 30
AND support_tickets > 3
AND plan = 'basic'
THEN flag as "likely to churn"
```
You, the human, decided the thresholds. Thirty days. Three tickets. You picked them from experience, and they're frozen in place until you change them by hand.
**What ML actually is.** Machine learning flips that around. Instead of *you* writing the rules, you hand the computer a pile of **historical examples** - past customers, each one labeled with what actually happened ("this one churned," "this one stayed") - and a learning procedure figures out the patterns on its own. The output of that process is a **model**: a thing you can feed a *new* customer's data and get back a prediction.
```mermaid
flowchart LR
hist[Historical examples
customers + what happened] -->|learning procedure| model[A model
learned patterns]
new[New customer] --> model
model --> pred["Prediction
'82% likely to churn'"]
```
π **Terminology.** A *model* is the learned thing - the output of training. It's not a program someone wrote line by line; it's a set of patterns the learning procedure distilled from your data. You feed it new inputs, it returns predictions.
**Why people get this wrong.** The common picture is that ML "understands" your customers, the way a person would. It doesn't - it found statistical patterns, correlations between inputs and outcomes. That's powerful, but it means the model is only as good as the examples. Give it a thousand churned customers who all happened to be on the basic plan, and it will happily learn "basic plan = churn" whether or not that's truly *why* they left. Hold onto that - it's the seed of every ML failure we'll cover later.
**Why this matters to you.** The "pile of historical examples" *is data work* - pulling, cleaning, and labeling it correctly is the job you already do. ML didn't replace the data person; it gave them a new place to be essential.
## Supervised learning: learning from labeled examples
The churn example above is the most common flavor of ML, called **supervised learning**. The "supervision" is the answer key: every historical example comes with the correct outcome attached.
π **Terminology.** A *label* (also called the *target*) is the answer you want to predict, recorded for each historical example. For churn, the label is "did this customer cancel - yes or no?" The columns you predict *from* are called **features** (we'll dig into those next phase).
```text
A supervised training table - every row has its answer
ββββββββββββββ¬βββββββββββββββ¬βββββββββββ¬βββββββββββββββ
β tenure_mo β tickets_90d β plan β churned? β β the LABEL
ββββββββββββββΌβββββββββββββββΌβββββββββββΌβββββββββββββββ€
β 24 β 0 β pro β no β
β 2 β 5 β basic β yes β
β 11 β 1 β pro β no β
β 3 β 4 β basic β yes β
ββββββββββββββ΄βββββββββββββββ΄βββββββββββ΄βββββββββββββββ
βββββββββββββ features βββββββββββββββ
```
The model studies the relationship between the feature columns and that final label column. Then, for a brand-new customer where you *don't* know the answer yet, it predicts the label.
Supervised learning splits into two everyday shapes, and the only difference is what kind of answer you're predicting:
- **Classification** - the label is a category. "Churn / no churn." "Spam / not spam." "This transaction is fraud / legitimate." The model outputs a class (often with a probability attached, like "82% likely to churn").
- **Regression** - the label is a number. "How many dollars will this customer spend next month?" "What will the order volume be on Friday?" The model outputs a number.
π‘ **Key point.** If you have historical examples *with known answers*, and you want to predict that answer for new cases, you're looking at supervised learning. Category answer β classification. Numeric answer β regression. That single distinction covers a huge fraction of real-world ML.
**A real example.** Here's what using a trained churn classifier looks like in practice - the details vary by tool, but the shape is universal:
```console
$ python predict_churn.py --customer 88213
Loading model: churn_classifier_v3.pkl
Customer 88213 tenure=3mo tickets_90d=4 plan=basic
Prediction: CHURN (probability 0.82)
```
*What just happened:* You handed the model one new customer's features. It compared them against the patterns it learned from thousands of past customers and returned a prediction - "churn" - with a confidence of 0.82, meaning the patterns most resemble customers who left. Nobody wrote an `IF tickets > 3` rule. The model derived its own version of that logic from the data, weighing all the features together.
β οΈ **A probability is not a fact.** That `0.82` is the model's estimate based on past patterns, not a guarantee this person will cancel. Treating model probabilities as certainties is one of the fastest ways to lose trust in a project. They're decision aids, not crystal balls.
## Unsupervised learning: finding structure with no answer key
Now flip the setup: a pile of customer data with *no labels at all* - nobody's told you who's "good" or "at risk." Just rows.
**What it actually is.** **Unsupervised learning** looks for structure that's already sitting in the data, without any answer key to aim at. The most common job here is **clustering**: grouping rows that resemble each other.
```text
Clustering: no labels going in - the algorithm
finds natural groupings on its own
spend
β β’ β’ β β β β three natural
β β’ β’ β’ β β β β clusters emerge
β β’ β’ β β β from the data,
β nobody labeled them
β β β β
β β β β β
β β β β
βββββββββββββββββββββββββββΊ engagement
```
You might run clustering on your customers and discover three natural groups - say, "high-spend, high-engagement," "new and tentative," and "dormant." The algorithm didn't know those names; *you* look at the groups afterward and interpret them. It just found that the rows fall into clusters.
**Why people get this wrong.** Because there's no answer key, there's no single "correct" output to check against. Two reasonable clustering runs can produce different groupings, and neither is "wrong." Unsupervised results are a starting point for human interpretation - not a verdict. If a teammate presents clusters as objective truth, that's a flag worth a gentle question.
**Where you'll meet it.** Customer segmentation, anomaly detection, and exploratory "what's even in this data?" work. Genuinely useful - just remember it answers "what structure is here?" not "what will happen?"
## The two side by side
| | Supervised | Unsupervised |
|---|---|---|
| **You haveβ¦** | examples *with* known answers (labels) | examples with *no* labels |
| **It doesβ¦** | predicts the answer for new cases | finds structure/groups in what you have |
| **Everyday jobs** | churn prediction, fraud detection, sales forecasting | customer segmentation, anomaly detection |
| **How you check it** | compare predictions to known answers | interpret the groups by hand; no single "right" |
Most ML you'll encounter at work - and everything in the next phase - is supervised, because most business questions are "what will happen?" and we usually have history to learn from.
## Recap
1. ML's core move: **learn patterns from historical examples** instead of you hand-writing the rules. The learned thing is a **model**.
2. The model is only as good as the examples - it finds correlations, it doesn't truly "understand."
3. **Supervised** learning predicts a known kind of answer (the **label**) for new cases - a category (**classification**) or a number (**regression**).
4. **Unsupervised** learning finds structure (like **clusters**) when there's no answer key - useful, but for interpretation, not prediction.
5. The "pile of examples" is data work - which is exactly why you belong in this conversation.
Next: the actual workflow of a supervised project - features, splitting the data so you can trust your results, training, and the subtle business of measuring whether the model is any good.
---
# The Workflow
Here's how a supervised project actually moves from your tables to a working model. There are four beats - **features, split, train, evaluate** - and a data person has a strong hand in three of them. The one piece that's "the algorithm" is, in practice, the smallest part.
```mermaid
flowchart LR
raw[Your raw tables] --> F[Features
shape the inputs]
F --> S[Split
train / test]
S --> T[Train
on the train set]
T --> E[Evaluate
on the test set]
```
## Features: the columns the model learns from
**What they actually are.** **Features** are the input columns you give the model to learn from - the things it's allowed to look at when making a prediction. If the label is "did this customer churn?", the features are everything *about* the customer the model gets to see: tenure, support tickets, plan type, last login date.
π **Terminology.** A *feature* is one input column. *Feature engineering* is the craft of turning your raw data into features that actually carry signal - and it's where data people quietly win or lose projects.
**Why this is more than "pick some columns."** Raw data is rarely in the right shape. A `last_login` timestamp isn't directly useful, but `days_since_last_login` is. A pile of support tickets isn't a feature, but `tickets_in_last_90_days` is. You're translating messy reality into clean, comparable numbers the model can reason over.
```text
Feature engineering: raw β useful
RAW ENGINEERED FEATURE
last_login = 2026-04-02 βββΊ days_since_last_login = 78
[ticket, ticket, ticket] βββΊ tickets_in_last_90_days = 3
signup = 2024-01-15 βββΊ tenure_months = 29
```
π‘ **Key point.** Models don't see your business; they see your features. A mediocre algorithm with thoughtful features usually beats a fancy algorithm with lazy ones. This is the single biggest lever a data person controls.
β οΈ **Don't sneak the answer into the features.** If one of your "features" is actually a stand-in for the outcome - for example, a `cancellation_reason` column that only gets filled in *after* someone churns - the model will look brilliant in testing and fall apart in real life. This is **data leakage**, and it gets full treatment in Phase 3. For now, plant one flag: a feature must be something you'd genuinely know *before* the outcome happens.
## The split: train on some, test on the rest - and why
Here's the idea that separates trustworthy ML from self-deception. Before training, you **split your labeled data into two parts**:
- a **training set** - the examples the model learns from, and
- a **test set** - examples you *hide* from the model during training, and use only at the end to check how it does.
```text
ALL your labeled data
βββββββββββββββββββββββββββββββββββββββββββββββββ
βββββββββββββββββββββββββββββββββββββββββββββββββββ
βββββββββββββββββββββββββββββββββββββββββββββββββ
βββββββββ TRAIN (model learns) ββββββ TEST βββββ
(hidden until
the very end)
```
**Why bother - why not train on everything?** Because the question that matters is *"how will this model do on customers it has never seen?"* Those are the only customers you'll ever use it on. Test it on the same examples it learned from and you're asking it to recite answers it already memorized - it'll look great and tell you nothing.
πͺ **War story.** The classic rookie result: a model scores beautifully in development, ships, then flops in production. Nine times out of ten, somebody evaluated on data the model had already seen, or the test data was contaminated by the training data. The split exists to catch this *before* it embarrasses you.
**A real example.** Doing the split is usually a couple of lines, but the *intent* is the whole game:
```console
$ python train.py
Loaded 50,000 labeled customers
Splitting: 40,000 train / 10,000 test (test set held out)
Training on 40,000 examples... done
Evaluating on the 10,000 held-out customers the model never saw...
```
*What just happened:* The model learned only from the 40,000 training customers. The 10,000 test customers stayed locked away until training finished, then got used to ask the real question: "on people you've never met, how do you do?" That number is the one you can trust.
β οΈ **The test set is sacred - look at it once.** If you peek at the test results, tweak the model to do better on them, peek again, tweak again, you've quietly turned your test set into a training set. Its validity leaks away with every peek. (A third "validation" slice handles tuning properly - detail for later; the principle stands regardless.)
## Training: the part that's mostly not your job
This is the step everyone pictures when they hear "machine learning," and it's genuinely the *least* hands-on for a data person. You hand the training set to a learning algorithm, and it adjusts itself to fit the patterns between features and label.
You don't need to know the internals to be useful here. What matters is the framing: **training is the algorithm's job; preparing what it trains on is yours.** Picking which algorithm is often a few lines of code and some experimentation. Getting clean, leak-free, well-featured data into it is the work that takes real judgment.
## Evaluating: accuracy is not the whole story
The model's trained. Now the crucial question: *is it any good?* The tempting answer is **accuracy** - what fraction of predictions were correct. It's fine to glance at, but on its own it can be dangerously misleading. Here's the trap:
**The rare-event problem.** Imagine fraud detection: genuine fraud is rare, most transactions are legitimate. Picture a lazy "model" that *always* predicts "not fraud":
```text
Suppose fraud is rare and most transactions are legit.
A model that ALWAYS says "not fraud":
β correct on every legitimate transaction
β wrong on every actual fraud
Result: very high accuracy...
...and it catches ZERO fraud. Completely useless.
```
That model can post impressive-sounding accuracy and still be worthless, because it never catches the thing you actually care about. (Exactly how high depends on how rare fraud is in *your* data - the point is that high accuracy can coexist with catching nothing.)
Use two sharper questions instead. Don't memorize formulas - hold the meaning:
π **Terminology.**
- **Precision** - *of the cases the model flagged, how many were real?* High precision means few false alarms. "When it cries fraud, it's usually right."
- **Recall** - *of all the real cases out there, how many did the model catch?* High recall means few misses. "It rarely lets real fraud slip through."
```text
the truth
fraud not fraud
model ββββββββββββ¬βββββββββββββ
says β caught β false β PRECISION = of all the
"fraud" β β (good) β alarm β "fraud" calls, how many
ββββββββββββΌβββββββββββββ€ were truly fraud?
model β MISSED β correctly β
"not β β (bad) β ignored β RECALL = of all the real
fraud" ββββββββββββ΄βββββββββββββ fraud, how much did we catch?
```
**The tension you can't escape.** Precision and recall pull against each other. Flag aggressively and you catch more fraud (high recall) but raise more false alarms (lower precision). Flag cautiously and your alarms are usually right (high precision) but you miss more real cases (lower recall). There's no universal "correct" balance - it depends on the cost of each mistake.
π‘ **Key point.** "Is the model good?" is a *business* question disguised as a technical one. For fraud, a missed case (low recall) might cost far more than a false alarm, so you'd lean toward recall. For flagging customers with a retention offer, too many false alarms annoy good customers, so you might lean toward precision. The right metric comes from "what does a mistake cost us?" - a conversation a data person should be *in*, not handed the answer to.
## Recap
1. **Features** are the input columns the model learns from; shaping raw data into good features (**feature engineering**) is high-leverage data work.
2. A feature must be something you'd know *before* the outcome - or you've got leakage (Phase 3).
3. **Split** into a **training set** and a hidden **test set** so you can measure on unseen data - the only reliable measure of real-world performance.
4. The **test set is sacred**: judge yourself on it once; don't tune against it.
5. **Training** is mostly the algorithm's job; preparing its data is yours.
6. **Accuracy alone misleads on rare events.** Use **precision** (few false alarms) and **recall** (few misses), and choose the balance from what a mistake actually costs.
Next: why ML projects actually succeed or fail on the data - and why that puts you closer to the center than the people writing the models.
---
# Where Data People Fit
Here's the truth practitioners learn the hard way, the one nobody puts on the recruiting slides: **machine learning lives or dies on the data.** The model is the easy part. The hard part is the part you already do.
If you take one thing from this guide, take this: the bottleneck in real ML is almost never "we need a smarter algorithm." It's "our data is messy, our features are weak, our pipeline broke, or something leaked." Every one of those is a data problem - not a consolation prize, the main event, and you're already standing on it.
## Garbage in, garbage out
β οΈ **Garbage in, garbage out.** A model learns whatever patterns are in the data you feed it - including the wrong ones. It cannot tell the difference between a real signal and a data-entry mistake. Feed it dirty data and it will faithfully, confidently learn nonsense.
This phrase is old because it's true, and ML makes it sharper than ever. Consider what "garbage" quietly looks like in the data you handle:
- A column where "unknown" was recorded as `0`, so the model treats missing-info customers as if they scored zero.
- A `country` field that's `"US"` in one system and `"United States"` in another - to the model, two different countries.
- Duplicated rows from a bad join, so some customers are silently counted three times and the model over-weights them.
- A label that's wrong - someone marked "churned" on customers who actually just switched plans.
None of these throw an error. The pipeline runs, the model trains, a number comes out. It's just *quietly wrong* - the most dangerous kind. Spotting these is precisely the skill you've built staring at real tables.
π‘ **Key point.** A model can't be better than its data. No algorithm, however fancy, recovers signal that the data never contained or fixes labels that were wrong. Clean inputs aren't a prerequisite to the "real" ML work - they *are* the real ML work.
## Data leakage: when the model peeks at the answer
This is the subtle killer that fools smart teams, flagged briefly in Phase 2 - and the failure most likely to bite *you* specifically.
π **Terminology.** *Data leakage* is when information that wouldn't really be available at prediction time sneaks into the features the model trains on. In effect, the model gets to peek at the answer - so it looks brilliant in testing and collapses in production.
**What it does in real life.** Remember the churn model. Suppose one of your feature columns is `account_closed_date`. It seems innocent - it's just a date. But that date only exists for customers who *already churned*. The model quickly discovers "if `account_closed_date` is filled in, this customer churned" - which is true, and completely useless, because at the moment you actually need a prediction, that field is empty. You're trying to predict churn *before* it happens, and you accidentally handed the model a column that's only populated *after* it happens.
```text
LEAKAGE: a feature that only exists once you know the answer
prediction time outcome happens
(what you really know) (the future)
β β
β features should β β account_closed_date
β come from HERE β gets filled in HERE
βΌ βΌ
ββββββΌβββββββββββββββββββββββββββΌββββββΊ time
β β
ββ if a "feature" is β
secretly from the right β
side, it has LEAKED βββββ
```
β οΈ **The model is peeking at the answer.** Leakage doesn't announce itself. The symptom is a model that performs *suspiciously* well - far better than the problem should allow. When results look too good, your first suspicion should be leakage, not genius. Ask of every feature: *"Would I genuinely have this value at the moment I need to predict?"* If the real answer is no, it leaks.
πͺ **War story.** Leakage hides in the most ordinary-looking places - timestamps that postdate the outcome, an ID that encodes when a record was created, an aggregate accidentally computed over the whole dataset (including the test rows) before the split. The person most likely to catch it is whoever understands where each column *comes from* and *when it's populated*. That's the data person. Not the modeler. You.
## Leak-free splits and reliable pipelines
Two more places where the data person is the last line of defense:
**Leak-free splits.** Phase 2's train/test split only protects you if it's done cleanly. A common slip: computing something across *all* the data - an average, a normalization, a fill-in value - *before* splitting, so information from the test rows bleeds into the training step. The split has to come first, and the test set must stay genuinely untouched. Getting this right is data discipline, not modeling cleverness.
**Reliable pipelines.** A model in production is fed by a **pipeline** - the plumbing that pulls data, cleans it, builds features, and delivers them to the model, over and over, automatically. If that plumbing silently changes - a source table renames a column, an upstream job starts emitting nulls, a currency switches from dollars to cents - the model keeps producing predictions, just *wrong* ones, with no error and no warning.
β οΈ **A model that's still running is not the same as a model that's still right.** Models degrade quietly when their input data drifts. The only defense: watch your data, validate it, alert when it changes - see [Data Quality and Observability](/guides/data-quality-and-observability), because in production that *is* the ML work.
## So where do you fit? Closer to the center than you think
Step back and look at the whole workflow with clear eyes:
```text
THE REAL WEIGHTING OF AN ML PROJECT
data sourcing & cleaning ββββββββββββββββ
feature engineering ββββββββββββ
leak-free splitting ββββββ
the model / algorithm βββ
monitoring the pipeline ββββββββββββββββ
(illustrative - proportions vary by project, but the
shape is real: the data work dwarfs the modeling)
```
Notice where "the model / algorithm" sits. The glamorous part is the smallest part. Everything large is data work - sourcing, cleaning, shaping, guarding, watching. Those are *your* skills, and they're the ones a project can't survive without.
π‘ **Key point.** You don't need to be a mathematician to be essential to ML. Bring clean inputs, thoughtful features, trustworthy splits, and reliable pipelines - and be the person who asks "wait, where does that column actually come from?" That question has saved more ML projects than any algorithm.
## Recap
1. ML **lives or dies on data** - the model is the easy part. The hard, decisive work is the data work you already do.
2. **Garbage in, garbage out**: a model faithfully learns whatever's in the data, including the mistakes - and they rarely throw errors.
3. **Data leakage** is the subtle killer: a feature that's only known *after* the outcome makes the model look brilliant in testing and useless in reality. Suspiciously good results mean *check for leakage*.
4. Keep splits **leak-free** (split before you compute anything across the data) and pipelines **reliable** (a running model can be silently wrong as its inputs drift).
5. The data person sits **closer to the center** of ML than the job titles suggest.
## Where to go from here
You now have the working model: ML learns from data, the workflow runs features β split β train β evaluate, and it all rests on data quality.
Neural networks, deep learning, and today's large language models are their own territory - watch for a future **ai-ml** category. Everything here about leakage, quality, and rigorous evaluation carries straight into it; the tools get bigger, the truth that data decides the outcome does not.
For now, double down on the foundations: [What Is Data Engineering](/guides/what-is-data-engineering) for the pipelines that feed ML, and [Data Quality and Observability](/guides/data-quality-and-observability) for keeping those inputs trustworthy.