# When It Has to Run Itself: Pipelines You've got a clean database and a query that answers your question perfectly. There's just one human left in the loop, and it's you - opening your laptop every Monday to run that query, format the result, and send it on. It works. Until the Monday you're sick, or busy, or you fat-finger the date and ship a wrong number to the whole team. That last human step is the pain this phase is about. When the work must happen *reliably, on a schedule, without anyone remembering to do it*, you've reached the third stage: you build a **pipeline.** ## The mental model: a recipe the computer follows for you πŸ“ **Terminology.** A **pipeline** is a defined sequence of steps - get the data, transform it, store or send the result - written down as code or configuration so a computer can run it automatically, the same way every time. Remember the "every Monday, by hand" list from Phase 1? A pipeline is that exact list, except it's written down precisely enough that a machine can follow it - and then set to run on its own. ```text In your head (Phase 1): As a pipeline (Phase 3): ───────────────────── ──────────────────────── "download the export" β†’ 1. pull this week's orders ┐ "clean the dates" β†’ 2. transform / clean β”‚ written as code, "run the totals" β†’ 3. run the summary query β”œβ”€ runs automatically "email the result" β†’ 4. write result + notify β”˜ every Monday 6am (you, remembering) β†’ a scheduler, never forgetting ``` *What just happened:* The process moved out of your head and into something durable. The steps are no longer a habit you perform - they're an artifact that exists, can be read, can be fixed, and runs whether or not you're awake. That single shift is the whole idea of a pipeline. πŸ“ **Terminology.** A **scheduler** is the piece that triggers the pipeline on a timetable - "every day at 6am," "every hour," "every Monday." It's the part that replaces *you remembering*. ## What a pipeline buys you ### Reliability A human doing manual steps will eventually skip one, or do it slightly differently, or not be there at all. A pipeline does the identical steps every run, and tells you when something goes wrong instead of quietly shipping a bad result. ```text Manual: works βœ“ works βœ“ YOU'RE ON VACATION βœ— works βœ“ typo'd the date βœ— Pipeline: works βœ“ works βœ“ works βœ“ works βœ“ works βœ“ ``` *What just happened:* The gaps and the slip-ups - the failures that come from a person being in the loop - are designed out. The pipeline isn't smarter than you; it's just tireless and consistent in a way no human can be every single week. ### Reproducibility Because the steps are written down as code, the process is no longer a mystery only you can perform. Anyone can read exactly what happens, run it again and get the same answer, and trace any number back to the steps that produced it. *What just happened:* This is the cure for Phase 1's deepest problem - the work that lived only in your head. Now "how was this number calculated?" has a real answer you can point at, and "can you run it again?" is a yes, not a panic. ### Scale A pipeline doesn't get tired or bored, so the things that made manual work impractical stop mattering. Run it hourly instead of weekly. Process a hundred files instead of one. Feed ten reports instead of one. The marginal cost of "do it again" drops to almost nothing. **Why this saves you later.** Every "can we also do this every day / for every region / for every client?" request stops being "that's a lot more of my time" and becomes "change one setting." The work scales without your hours scaling with it. ## What a pipeline costs Here's the plain truth of the other side, and it's the whole reason you don't start here. A pipeline is **software**, and software has to be built and looked after. - **Up-front engineering.** Someone has to write the steps as code, connect the pieces, and set up the schedule. That's real work, and it's slower than just doing the task by hand once. - **It can break in new ways.** The source format changes, a server is down, credentials expire - and now there's a *system* that can fail, not just a task you forgot. You need it to alert you when it does. - **Ongoing maintenance.** Pipelines need monitoring, occasional fixing, and updating as the data around them changes. You've traded "I redo this every week" for "I keep this running." ```text THE TRADE ───────── manual: cheap to start, expensive forever (your time, every single run) pipeline: costly to build, cheap to run (engineering up front, then it runs itself) ``` *What just happened:* You're not getting automation for free - you're paying for it once, up front, in engineering, in exchange for paying almost nothing per run afterward. Whether that's a good deal depends entirely on how often the work repeats and how much it matters when it's wrong. ## ⚠️ Match the tool to the scale - don't over-engineer This is the most important judgment call in the whole guide: **a pipeline is the right answer only when the work genuinely repeats and genuinely matters.** Building one for a task you'll do twice is a classic, expensive mistake - you'll spend days automating something a five-minute manual job would have handled, then spend more days maintaining it. A rough guide to where each stage earns its keep: | The work is… | Reach for… | |---|---| | One-time, exploratory, small, just you | A **spreadsheet** | | Shared, growing, queried often, multi-person | A **database + SQL** | | Repeating on a schedule, must be reliable, no human in the loop | A **pipeline** | The plain rule of thumb: do it by hand until the manual cost clearly outweighs the cost of automating it. Let the pain make the case. A task you do once a quarter rarely justifies a pipeline; a report ten people depend on every morning almost always does. ## Where this goes next This phase is the *when* and *why* of pipelines. The *how* - the patterns for actually building them, the difference between extracting-then-transforming and the reverse, the tools people use - is a guide of its own: [ETL & ELT Pipelines](/guides/etl-elt-pipelines). When the trade-off above tips toward "yes, build it," that's where you go to build it well. And if you found yourself nodding along to this whole progression - spreadsheet to database to automated pipeline - you've just described the heart of a whole discipline. That field has a name and a map: [What Is Data Engineering](/guides/what-is-data-engineering). ## Recap 1. A **pipeline** is your manual "every Monday" steps written as code and run automatically by a **scheduler** - the process moved out of your head and into something durable. 2. It buys you **reliability** (same steps, every time, no skipped weeks), **reproducibility** (anyone can read and rerun it), and **scale** (do it hourly, for everything, without more of your hours). 3. It costs **up-front engineering**, **new ways to break**, and **ongoing maintenance** - automation is paid for once, in advance. 4. **Match the tool to the scale.** Spreadsheet for one-time work, database for shared querying, pipeline only when the work truly repeats and truly matters. Don't over-engineer. You now have the whole arc. The next time a data task lands on your desk, you can place it - and pick the tool that fits, not the one that's fanciest.