# Spreadsheets → SQL → Pipelines > The natural progression most data work actually follows - start in a spreadsheet, graduate to SQL when the sheet breaks, and build a pipeline when the work has to run itself. --- # Spreadsheets → SQL → Pipelines You started in a spreadsheet. Almost everyone does. Then one day the file got slow, or a teammate overwrote your numbers, or you realized you'd been hand-copying the same report every Monday for a year. That nagging feeling - *there has to be a better way to do this* - is real, and it's the moment your data work is ready to grow. This guide walks the path most data work actually travels: **spreadsheet → SQL → pipeline.** Three stages, each solving a specific pain the stage before it couldn't. You don't skip ahead because a tool is fancier; you move up only when the work outgrows where it lives. By the end you'll be able to look at a messy data task and say, calmly, "this belongs in a spreadsheet" - or "this needs a database now" - or "this has to become a pipeline." ## How to read this - **Trying to decide what tool fits a specific task?** Skim each phase's opening - every phase names the exact pain that signals "time to move up." - **Want it to finally make sense?** Read in order. Each stage is built on the one before, and the whole point is seeing *why* you move, not just *that* you do. ## The phases 1. **[Where Everyone Starts: Spreadsheets](01-where-everyone-starts-spreadsheets.md)** - why spreadsheets are genuinely great, and the exact places they quietly break. 2. **[Outgrowing the Sheet: SQL & Databases](02-outgrowing-the-sheet-sql-and-databases.md)** - when one shared source of truth, real types, and millions of rows mean it's time to graduate. 3. **[When It Has to Run Itself: Pipelines](03-when-it-has-to-run-itself-pipelines.md)** - when the work must be automated, scheduled, and repeatable, you build a pipeline - and what that buys and costs. > This guide is about *when* to move and *why*. The deep mechanics of building pipelines live in their > own guide: [ETL & ELT Pipelines](/guides/etl-elt-pipelines). And the broader field this all rolls up > into is covered in [What Is Data Engineering](/guides/what-is-data-engineering). --- # Where Everyone Starts: Spreadsheets Open a spreadsheet and there's your data, right in front of you - a grid you can click, sort, color, and fix on the spot. No setup, no login, no waiting. This is why it's the most-used data tool on Earth - starting there isn't a beginner's mistake. For a huge amount of real work, a spreadsheet is exactly the right answer. The skill that actually matters isn't avoiding spreadsheets. It's knowing the precise moments they stop serving you - so you can move up *before* a deadline or a broken report forces your hand. ## Why spreadsheets are genuinely great **You can see everything.** The data is the interface. There's no hidden layer between you and the numbers - what you see is what you've got. For exploring a new dataset, spotting an obvious outlier, or showing a colleague "look, here," nothing beats it. **They're endlessly flexible.** A cell can hold a number, a word, a date, a formula, or a note to yourself. You can restructure on a whim - insert a column, jot a comment, paste a chart next to the table. The tool never tells you "no." **They're instant.** Type a formula, see the answer. Change a number, watch everything recalculate. That tight feedback loop is genuinely powerful for thinking through a problem. 💡 **Key point.** A spreadsheet's strengths - visible, flexible, instant - are exactly what make it the right tool for small, exploratory, one-person, one-time work. Hold onto that. It's the same yardstick you'll use to know when you've outgrown it. ## Where spreadsheets quietly break None of the following means spreadsheets are bad. It means they have a *range*, and past that range the very flexibility that helped you starts working against you. ### Size limits A spreadsheet keeps everything in memory and re-runs every formula on every change. That's why it feels instant at small sizes - and why it crawls, then chokes, as rows pile up. There's even a hard ceiling. ```text Excel: max 1,048,576 rows per sheet (source: Microsoft docs) Google Sheets: max 10,000,000 cells per file (source: Google docs) ``` *What just happened:* Those aren't "slows down around here" numbers - they're walls. Hit them and the tool refuses more data. You'll feel pain long before the wall, too: a sheet with a few hundred thousand rows and a column of lookups can take seconds to recalc on every edit. ### No real types In a database, a column is *declared* - "this column holds dates, and nothing else." A spreadsheet makes no such promise. Every cell decides its own type, and the tool guesses. ```text A column you think is "dates": 2026-03-01 ← stored as a date 3/1/2026 ← stored as a date, different format March 1 ← stored as TEXT (no year, can't sort) '2026-03-01 ← leading apostrophe → stored as TEXT ``` *What just happened:* Four cells that all look like the same date to you are four different things to the spreadsheet. Sort that column and the text rows scatter to the wrong place. Sum a "number" column where one cell is secretly text, and it silently leaves that row out - no guard rail, because there's no declared type. ⚠️ **The gotcha that eats real data.** Spreadsheets auto-convert anything that *looks* like a number or date. A gene name like `SEPT2` becomes the date "September 2." A part number `00123` loses its leading zeros and becomes `123`. This is so common that scientists renamed human genes to stop spreadsheets from corrupting them (source: HUGO Gene Nomenclature Committee, 2020). If your IDs or codes matter exactly as written, the spreadsheet is silently rewriting them. ### Copy-paste errors Because the data and the formulas live in the same grid, one stray paste or one dragged-too-far formula can quietly break a calculation - and there's nothing to catch it. A famous economics paper's conclusions were undone when a formula was found to have skipped five rows of a selected range (source: Herndon, Ash & Pollin, 2013). The error wasn't exotic. It was a range that didn't cover all the data - the most ordinary spreadsheet mistake there is. ```text =AVERAGE(B2:B15) ← you meant B2:B20, dragged the box too short → the answer is wrong, and nothing warns you ``` *What just happened:* The formula ran perfectly. It averaged exactly what you told it to - which wasn't what you meant. Spreadsheets do exactly as asked, and "as asked" is easy to get subtly wrong by hand. ### No single source of truth The moment a spreadsheet matters, it multiplies. Someone emails `budget_final.xlsx`. Someone replies with `budget_final_v2.xlsx`. A third person edits the copy in their Downloads folder. Now there are three "truths" and no way to know which is current. ```text budget.xlsx budget_final.xlsx budget_final_v2.xlsx budget_final_USE_THIS_ONE.xlsx ← which number is real? ``` *What just happened:* Each copy drifted independently. With files, "the latest version" is a hope, not a guarantee. Shared cloud sheets help - one file, many editors - but they trade the version problem for a new one: two people editing the same cell, with the last save quietly winning. ### Manual means unrepeatable This is the deepest one. The work in a spreadsheet lives partly in the file and partly in *your head* - the steps you did by hand. "Paste the new data here, delete the blank rows, fix the date column, drag the formula down, refresh the chart." Next month you have to remember and redo all of it, perfectly. ```text Every Monday, by hand: 1. download this week's export 2. paste into the sheet 3. clean the date column 4. drag formulas down 5. update the chart range 6. email the result ← miss one step, or do it slightly differently, and the report is wrong ``` *What just happened:* Nothing was recorded. The process exists only as a habit, which means it can't be trusted to run the same way twice, can't be handed to a teammate cleanly, and stops entirely the week you're on vacation. A spreadsheet stores *results*, not *the steps that produced them*. ## So when do you move up? You move up when the work crosses one of those lines - not before. As a quick read: | The pain you're feeling | What it's telling you | |---|---| | The file is slow, or you're near the row limit | Size has outgrown the sheet | | IDs or dates keep getting mangled | You need real, declared types | | Numbers don't match and nobody knows which file is right | You need one shared source of truth | | You redo the same steps every week, by hand | The work needs to be automated | The first three point you toward a **database and SQL** - Phase 2. The last one, once a database is in place, points toward a **pipeline** - Phase 3. We'll take them in that order, because that's the order the pain usually arrives. ## Recap 1. Spreadsheets win at being **visible, flexible, and instant** - perfect for small, exploratory, one-person work. 2. They break on **size** (real, hard row/cell limits), **no real types** (cells guess, and auto-convert your IDs and dates), and **copy-paste errors** (formulas do exactly as asked, which is easy to get wrong). 3. They break on **no single source of truth** (files multiply and drift) and on being **manual** (the steps live in your head, so the work can't be repeated reliably). 4. You graduate when the work crosses one of those lines - and the *which line* tells you *where* to go next. Next: the tool built precisely for the first three pains - a database, and the language you ask it questions in. --- # Outgrowing the Sheet: SQL & Databases You hit one of the walls from Phase 1 - the file got slow, the IDs got mangled, or three "final" versions disagreed. That's not a failure. It's the signal that your data has grown up and needs a home built for the job. That home is a **database**, and the language you use to talk to it is **SQL**. Here's the reassuring part: a database isn't a strange new world, it's a spreadsheet's ideas made sturdy. If you understand a sheet, you're most of the way to understanding a table. ## The mental model: a table is a sheet that keeps its promises 📝 **Terminology.** A **database** is an organized store of data, managed by software built to keep it correct and fast. A **table** is one collection of data inside it - rows and columns, like a single spreadsheet tab. **SQL** (Structured Query Language) is how you ask the database to fetch, filter, or change data. The shapes map almost one-to-one: ```text SPREADSHEET DATABASE ─────────── ──────── a sheet/tab → a table a row → a row (one record: one order, one user) a column → a column (but with a DECLARED type) a cell → a value a formula / VLOOKUP → a SQL query (asked of the whole table) ``` *What just happened:* The pieces are the same. The difference is everything around them. A table *declares* what each column holds and refuses anything that doesn't fit, the database is one shared store instead of a file you copy, and SQL lets you ask questions across millions of rows without dragging a formula anywhere. Each of those maps to a Phase 1 pain. ## One shared source of truth In Phase 1, files multiplied until nobody knew which number was real. A database fixes this at the root: there is *one* table, living in one place, and everyone connects to that same table. There's no copy in someone's Downloads folder, because there are no copies - only connections. ```text Spreadsheet world: Database world: you ── budget_v2.xlsx you ─┐ me ── budget_FINAL.xlsx me ─┼─→ ONE orders table her ── budget_USE_THIS.xlsx her ─┘ (everyone sees the same rows) ``` *What just happened:* The "which version is current?" question disappears, because there's only ever one version. Read the `orders` table and you're reading the live, current data - the same data your teammate sees at the same moment. Next time two reports disagree, there's no afternoon spent hunting for the "right" file - there's one table, and the answer is whatever it says right now. ## Real types and constraints In Phase 1, a "date" column was a mix of real dates and look-alike text, and IDs got auto-mangled. A database makes you *declare* each column's type up front - and then it enforces that promise on every single row, forever. ```text CREATE TABLE orders ( id INTEGER, ← whole numbers only customer TEXT, ← any text amount DECIMAL, ← money, kept exact ordered_at DATE ← real dates only; "March 1" with no year is rejected ); ``` *What just happened:* You told the database the rules once. Now if anything tries to put the word "soon" into `ordered_at`, the database refuses it instead of silently storing text that looks like a date. The guard rail the spreadsheet never had is now built into the column itself. 📝 **Terminology.** A **constraint** is a rule the database enforces on a column - for example, "this value can't be empty," or "every `id` must be unique." It's the database catching bad data at the door, rather than you finding it three weeks later in a broken report. The whole class of bug where one stray text cell silently breaks a `SUM`, or a part number loses its leading zeros, is simply gone - the wrong shape isn't allowed in. ## Querying millions of rows In Phase 1, the sheet slowed to a crawl and then hit a hard row ceiling. Databases are built for the opposite: engineered to filter, sort, and summarize enormous tables quickly, because that's their entire job. You don't drag a formula down a million rows - you write one sentence describing what you want, and the database figures out how to get it fast. A `SELECT` query is you saying "give me these columns, from this table, where these conditions hold." ```console SELECT customer, amount FROM orders WHERE amount > 100; ``` ```text customer amount ──────── ────── Ada 250.00 Grace 140.00 Lin 199.50 ``` *What just happened:* You asked one question - "show me the customer and amount for every order over 100" - and the database scanned the whole `orders` table and handed back only the matching rows. It does the same whether the table has a hundred rows or a hundred million, and it doesn't melt your laptop doing it. "The file is too big to open" stops being a sentence you say - bigger data is just a longer scan, not a wall. > The grammar of `SELECT … FROM … WHERE …` is the single most useful thing to learn here, and it has its > own guide: [Querying Basics: SELECT & WHERE](/guides/querying-basics-select-where). If this is your > first SQL query, read that next. ## Multiple users, safely A shared spreadsheet lets several people edit at once, but resolves clashes crudely - often the last save wins, quietly erasing someone else's change. Databases are designed from the start for many people and programs reading and writing at once without trampling each other. 📝 **Terminology.** A **transaction** is a group of changes the database treats as all-or-nothing: either every step succeeds, or none of them do. It's how a database moves money from one account to another without ever leaving it half-moved - even if two people press "go" at the same instant. *What just happened:* Where the spreadsheet's answer to "two people, same data" was "hope for the best," the database's answer is a set of rules that keep the data correct no matter how many people are touching it - the difference between a tool for one person and a tool for a team. As soon as more than one person, or one automated script, depends on your data, you need it to stay correct under simultaneous use. The database gives you that for free; the spreadsheet never could. ## A worked move: the same task, leveled up Picture the Phase 1 ritual: every Monday, opening a sheet of this week's orders and, by hand, filtering to the big ones and eyeballing the total. In a database, that becomes a question you *ask*: ```console SELECT customer, SUM(amount) AS total FROM orders WHERE ordered_at >= '2026-06-15' GROUP BY customer; ``` ```text customer total ──────── ────── Ada 490.00 Grace 140.00 ``` *What just happened:* One query did the filtering and the per-customer totaling in a single step - no dragging, no manual cleanup, reading the same live table everyone else does. Same result you used to assemble by hand, now one repeatable sentence. **Try it yourself.** Same idea on a tiny library dataset - join two tables and count, the kind of question that would mean a fragile VLOOKUP in a sheet. Run it and tweak the `WHERE`: ```sql runnable SELECT a.name AS author, a.country, COUNT(b.id) AS books FROM authors a JOIN books b ON b.author_id = a.id GROUP BY a.id ORDER BY books DESC; ``` And that word - *repeatable* - is the bridge to Phase 3. You've fixed size, types, and the source of truth, but you're still the one running this query every Monday. What happens when even *that* should run on its own? ## Recap 1. A **table** is a spreadsheet's ideas made sturdy: rows and columns, but with declared, enforced types. 2. A database gives you **one shared source of truth** - connections, not copies - so versions stop multiplying. 3. **Types and constraints** catch bad data at the door, killing the "stray text broke my SUM" class of bug. 4. Databases **query millions of rows** fast, and **SQL** (`SELECT … FROM … WHERE …`) is how you ask. 5. They handle **many users safely**, with transactions keeping data correct under simultaneous use. You can now hold a clean, shared, queryable source of truth. Next: what to do when the querying itself has to happen without you. --- # When It Has to Run Itself: Pipelines You've got a clean database and a query that answers your question perfectly. There's just one human left in the loop, and it's you - opening your laptop every Monday to run that query, format the result, and send it on. It works. Until the Monday you're sick, or busy, or you fat-finger the date and ship a wrong number to the whole team. That last human step is the pain this phase is about. When the work must happen *reliably, on a schedule, without anyone remembering to do it*, you've reached the third stage: you build a **pipeline.** ## The mental model: a recipe the computer follows for you 📝 **Terminology.** A **pipeline** is a defined sequence of steps - get the data, transform it, store or send the result - written down as code or configuration so a computer can run it automatically, the same way every time. Remember the "every Monday, by hand" list from Phase 1? A pipeline is that exact list, except it's written down precisely enough that a machine can follow it - and then set to run on its own. ```text In your head (Phase 1): As a pipeline (Phase 3): ───────────────────── ──────────────────────── "download the export" → 1. pull this week's orders ┐ "clean the dates" → 2. transform / clean │ written as code, "run the totals" → 3. run the summary query ├─ runs automatically "email the result" → 4. write result + notify ┘ every Monday 6am (you, remembering) → a scheduler, never forgetting ``` *What just happened:* The process moved out of your head and into something durable. The steps are no longer a habit you perform - they're an artifact that exists, can be read, can be fixed, and runs whether or not you're awake. That single shift is the whole idea of a pipeline. 📝 **Terminology.** A **scheduler** is the piece that triggers the pipeline on a timetable - "every day at 6am," "every hour," "every Monday." It's the part that replaces *you remembering*. ## What a pipeline buys you ### Reliability A human doing manual steps will eventually skip one, or do it slightly differently, or not be there at all. A pipeline does the identical steps every run, and tells you when something goes wrong instead of quietly shipping a bad result. ```text Manual: works ✓ works ✓ YOU'RE ON VACATION ✗ works ✓ typo'd the date ✗ Pipeline: works ✓ works ✓ works ✓ works ✓ works ✓ ``` *What just happened:* The gaps and the slip-ups - the failures that come from a person being in the loop - are designed out. The pipeline isn't smarter than you; it's just tireless and consistent in a way no human can be every single week. ### Reproducibility Because the steps are written down as code, the process is no longer a mystery only you can perform. Anyone can read exactly what happens, run it again and get the same answer, and trace any number back to the steps that produced it. *What just happened:* This is the cure for Phase 1's deepest problem - the work that lived only in your head. Now "how was this number calculated?" has a real answer you can point at, and "can you run it again?" is a yes, not a panic. ### Scale A pipeline doesn't get tired or bored, so the things that made manual work impractical stop mattering. Run it hourly instead of weekly. Process a hundred files instead of one. Feed ten reports instead of one. The marginal cost of "do it again" drops to almost nothing. **Why this saves you later.** Every "can we also do this every day / for every region / for every client?" request stops being "that's a lot more of my time" and becomes "change one setting." The work scales without your hours scaling with it. ## What a pipeline costs Here's the plain truth of the other side, and it's the whole reason you don't start here. A pipeline is **software**, and software has to be built and looked after. - **Up-front engineering.** Someone has to write the steps as code, connect the pieces, and set up the schedule. That's real work, and it's slower than just doing the task by hand once. - **It can break in new ways.** The source format changes, a server is down, credentials expire - and now there's a *system* that can fail, not just a task you forgot. You need it to alert you when it does. - **Ongoing maintenance.** Pipelines need monitoring, occasional fixing, and updating as the data around them changes. You've traded "I redo this every week" for "I keep this running." ```text THE TRADE ───────── manual: cheap to start, expensive forever (your time, every single run) pipeline: costly to build, cheap to run (engineering up front, then it runs itself) ``` *What just happened:* You're not getting automation for free - you're paying for it once, up front, in engineering, in exchange for paying almost nothing per run afterward. Whether that's a good deal depends entirely on how often the work repeats and how much it matters when it's wrong. ## ⚠️ Match the tool to the scale - don't over-engineer This is the most important judgment call in the whole guide: **a pipeline is the right answer only when the work genuinely repeats and genuinely matters.** Building one for a task you'll do twice is a classic, expensive mistake - you'll spend days automating something a five-minute manual job would have handled, then spend more days maintaining it. A rough guide to where each stage earns its keep: | The work is… | Reach for… | |---|---| | One-time, exploratory, small, just you | A **spreadsheet** | | Shared, growing, queried often, multi-person | A **database + SQL** | | Repeating on a schedule, must be reliable, no human in the loop | A **pipeline** | The plain rule of thumb: do it by hand until the manual cost clearly outweighs the cost of automating it. Let the pain make the case. A task you do once a quarter rarely justifies a pipeline; a report ten people depend on every morning almost always does. ## Where this goes next This phase is the *when* and *why* of pipelines. The *how* - the patterns for actually building them, the difference between extracting-then-transforming and the reverse, the tools people use - is a guide of its own: [ETL & ELT Pipelines](/guides/etl-elt-pipelines). When the trade-off above tips toward "yes, build it," that's where you go to build it well. And if you found yourself nodding along to this whole progression - spreadsheet to database to automated pipeline - you've just described the heart of a whole discipline. That field has a name and a map: [What Is Data Engineering](/guides/what-is-data-engineering). ## Recap 1. A **pipeline** is your manual "every Monday" steps written as code and run automatically by a **scheduler** - the process moved out of your head and into something durable. 2. It buys you **reliability** (same steps, every time, no skipped weeks), **reproducibility** (anyone can read and rerun it), and **scale** (do it hourly, for everything, without more of your hours). 3. It costs **up-front engineering**, **new ways to break**, and **ongoing maintenance** - automation is paid for once, in advance. 4. **Match the tool to the scale.** Spreadsheet for one-time work, database for shared querying, pipeline only when the work truly repeats and truly matters. Don't over-engineer. You now have the whole arc. The next time a data task lands on your desk, you can place it - and pick the tool that fits, not the one that's fanciest.