# Python From Zero

> Learn Python from nothing to genuinely advanced: install and the basics, then objects and tooling, then the deep half - the data model, generators, decorators, typing, concurrency and the GIL, performance, and packaging - all in small, runnable steps with clear explanations.


---

# Python From Zero

Python is the language people reach for when they want to *get something done* without fighting the
language first - data work, scripts, web backends, automation, glue between systems. It reads almost
like English, which makes it inviting, and that same readability hides a few sharp edges nobody warns
you about. This guide takes you the whole way: from "I've never run Python" to understanding what the
language is *actually doing* underneath your code - explaining each piece rather than handing you spells
to memorize.

It's one zero-to-hero journey in two halves. **Phases 1–9 are the basics** - enough to write real,
well-organized programs. **Phases 10–18 are the deep half** - the data model, generators, decorators,
typing, concurrency, performance, and packaging, the stuff that separates "writes Python" from
"understands Python." Each phase carries a difficulty badge so you can see the climb.

If programming itself is brand new - not just Python - start with
[Programming From Zero](/guides/programming-from-zero) first; it builds the "what is a program" mental
model this guide assumes.

## How to read this
- **Brand new to Python?** Read 1–9 in order, top to bottom - each builds on the last. Type the examples
  as you go (most are runnable right here); doing beats reading. Come back for 10+ when the basics feel
  comfortable.
- **Already know another language?** Skim 1–5 for Python's spelling of ideas you have, then start properly
  at [Phase 6: Objects & Classes](06-objects-and-classes.md).
- **Past the basics already?** Jump straight to the deep half - [Phase 10: The Data Model](10-the-data-model.md)
  onward is where Python stops being "a readable scripting language" and starts being a language you can
  reason about to the metal.

## The phases

**Part 1 - The basics (🟢 Basic → 🟡 Intermediate)**
1. **[Install & Your First Program](01-install-and-first-program.md)** 🟢 - Python 3, the REPL, a real `hello.py`.
2. **[Syntax, Values & Types](02-syntax-values-and-types.md)** 🟢 - indentation as structure, the core types, f-strings.
3. **[Collections](03-collections.md)** 🟢 - lists, tuples, dicts, sets, and the aliasing trap.
4. **[Control Flow & Functions](04-control-flow-and-functions.md)** 🟢 - `if`/loops/`def`, truthiness, the mutable-default trap.
5. **[Modules & Project Layout](05-modules-and-project-layout.md)** 🟢 - `import`, the stdlib, `__main__`, a clean layout.
6. **[Objects & Classes](06-objects-and-classes.md)** 🟡 - what an object really is, `self`, when to reach for a class.
7. **[Errors & I/O](07-errors-and-io.md)** 🟡 - reading tracebacks, `try`/`except`, files without corruption.
8. **[Ecosystem & Tooling](08-ecosystem-and-tooling.md)** 🟡 - `pip`, virtual environments, formatters, linters.
9. **[Idioms & Gotchas](09-idioms-and-gotchas.md)** 🟡 - the Pythonic way, and the sharp edges that bite everyone once.

**Part 2 - Beyond the basics (🔴 Advanced)**
10. **[The Data Model & Dunder Methods](10-the-data-model.md)** 🔴 - make your objects behave like built-ins.
11. **[Iterators & Generators](11-iterators-and-generators.md)** 🟡 - `yield`, laziness, processing huge data without the RAM.
12. **[Decorators](12-decorators.md)** 🔴 - the `@` magic in every framework, demystified.
13. **[Context Managers](13-context-managers.md)** 🟡 - `with`, and never leaking a file/lock/connection again.
14. **[Type Hints & mypy](14-type-hints.md)** 🟡 - gradual typing; catch the bug before runtime.
15. **[Dataclasses & Modern Modeling](15-dataclasses.md)** 🟡 - kill the boilerplate.
16. **[Concurrency & the GIL](16-concurrency-and-the-gil.md)** 🔴 - threads vs processes vs async, and what the GIL actually blocks.
17. **[Performance & Memory](17-performance-and-memory.md)** 🔴 - how CPython runs, the GC, and the real speedups.
18. **[Packaging & Environments](18-packaging-and-environments.md)** 🟡 - ship a package, not a folder of scripts.

**Finale**
19. **[Where to Go Next](19-where-to-go-next.md)** 🟢 - web, data, automation, and what to build.

> Frameworks (FastAPI, Django) are their own guides - they're different tools, not "more Python." This
> guide makes the *language* make sense, top to bottom.


---

# Install & Your First Program

To write Python, your computer needs the **interpreter** - the program that reads `.py` files and
executes them line by line. Installing it trips people up most, usually over a boring reason: the name.
Let's install it, prove it's there, and run real code two ways.

📝 **Terminology.** The **Python interpreter** is the program named `python` (or `python3`) that
executes your code. When people say "run it in Python," they mean "hand this file to the interpreter."

## Install Python 3

You want **Python 3** (3.13 or newer is great; anything 3.10+ is fine for this guide). Pick your
operating system:

**Windows** - Go to [python.org/downloads](https://www.python.org/downloads/), download the latest
Python 3 installer, and run it. On the very first screen, check **"Add python.exe to PATH"** before
clicking Install - that one checkbox prevents the most common beginner headache: the terminal not
finding Python afterward.

**macOS** - macOS ships an old, system-managed Python you shouldn't rely on. Install a fresh one with
[Homebrew](https://brew.sh):
```console
$ brew install python
```
*What just happened:* Homebrew installed an up-to-date Python 3 and wired up the `python3` command. No
Homebrew? The python.org installer works just as well.

**Linux (Debian/Ubuntu)** - Python 3 is usually already present, but install it explicitly to be sure:
```console
$ sudo apt update
$ sudo apt install python3
```
*What just happened:* `apt` (the system package manager) ensured `python3` is installed; `sudo` grants
the admin rights that installing system software requires.

## Confirm it actually installed

Open a fresh terminal (a new window, so it picks up the updated PATH) and ask Python its version:
```console
$ python3 --version
Python 3.14.0
```
*What just happened:* `--version` makes the interpreter print its version and exit instead of running
code. A `Python 3.x.x` line proves it's installed and reachable - your exact number may differ.

⚠️ **`python` vs `python3` - the name confusion that wastes everyone's first afternoon.** On macOS and
Linux, the command is almost always `python3` (plain `python` may not exist, or may point at an ancient
Python 2). On Windows, the installer typically sets up `python` (and `py`). If one name gives "command
not found," try the other:
```console
$ python --version
Python 3.14.0
```
This guide writes `python3` throughout. **On Windows, type `python` (or `py`) wherever you see
`python3`** - same interpreter, different spelling.

⚠️ **"command not found" / "not recognized."** If *neither* name works, the interpreter is installed but
your terminal can't find it - a PATH problem. On Windows, re-run the installer, check **"Add python.exe
to PATH"** (or choose "Modify" and enable it), then open a new terminal - PATH changes only apply to
terminals opened *afterward*.

## The REPL - a place to try one line at a time

Run `python3` with no file name to get the **REPL** - an interactive prompt where you type one line,
press Enter, and immediately see the result. It's the fastest way to test an idea.
```console
$ python3
Python 3.14.0 (main, Oct  7 2025, 12:00:00) [Clang 17.0.0] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> 2 + 2
4
>>> print("hi from the REPL")
hi from the REPL
>>>
```
*What just happened:* `>>>` is the REPL's prompt. You typed `2 + 2`; the interpreter evaluated it and
printed `4` back - no `print` needed, the REPL shows the value of whatever you type. It's a scratchpad:
try a snippet, see what it does, throw it away.

📝 **Terminology.** **REPL** stands for **R**ead-**E**val-**P**rint **L**oop: it *reads* your line,
*evaluates* it, *prints* the result, then *loops* back for the next one.

To leave the REPL and return to your normal terminal, type `exit()` and press Enter (or press
**Ctrl-D** on macOS/Linux, **Ctrl-Z** then Enter on Windows):
```console
>>> exit()
$
```
*What just happened:* `exit()` told the interpreter to quit; you're back at your shell's `$` prompt. The
REPL forgets everything on close - exactly why real programs live in files.

## Your first program in a file

A REPL is great for experiments, but real programs are saved files you can run again. Create a file
called `hello.py` in any folder, with one line in it:
```python runnable
print("Hello, Python!")
```
*What just happened:* The same `print` instruction from the REPL, now saved to disk. `print` displays
whatever's in the parentheses - here, the text in quotes. (New to `print` and "argument"? [Programming
From Zero, Phase 1](/guides/programming-from-zero) walks through them slowly.)

Now run the file by handing it to the interpreter:
```console
$ python3 hello.py
Hello, Python!
```
*What just happened:* `python3 hello.py` read the file and ran its instructions top to bottom, hit the
`print` line, and showed your text - a complete Python program, written, saved, and run.

⚠️ **`can't open file 'hello.py'`.** Seeing `python3: can't open file 'hello.py': [Errno 2] No such file
or directory` means your terminal isn't *in* the folder where you saved the file. `cd` into that folder
(e.g. `cd Desktop`) and run the command again - the interpreter looks for files relative to where your
terminal is "standing."

💡 **Key point.** Two ways to run Python, two jobs. The **REPL** (`python3` alone) tries one line at a
time and throws it away. A **file** (`python3 yourfile.py`) is for real, repeatable programs. You'll
live in files, keeping the REPL open on the side for quick tests.

## Recap

1. Install **Python 3** - python.org on Windows (check **Add to PATH**), Homebrew on macOS, `apt` on
   Linux.
2. **`python3 --version`** confirms it's installed and reachable. On Windows, use `python` or `py`.
3. The command name is the #1 gotcha: `python3` on macOS/Linux, usually `python` on Windows. "Not
   found" means a PATH problem - reinstall with PATH enabled, open a fresh terminal.
4. The **REPL** (`python3` with no file) runs one line at a time and prints results; `exit()` leaves it.
5. Save real code in a `.py` file and run it with **`python3 yourfile.py`**.

Next: values, the types they come in, and Python's use of *indentation* to structure code.


---

# Syntax, Values & Types

Most languages use curly braces `{}` and semicolons to mark code blocks. Python uses something you can't
see: **whitespace**. Getting it wrong produces mysterious errors until you know the rule, so it's worth
meeting head-on - then we'll cover the handful of value types Python is built from.

## The big surprise: indentation *is* structure

**What it actually is.** In Python, a line's *indentation* - how many spaces it's pushed in from the
left - groups lines into a block. Where other languages write `{ ... }`, Python says "these lines are
indented under that line, so they belong to it." Indentation isn't decoration; it's the grammar.

A block of code that runs only when a condition is true (`if` proper is [Phase
4](04-control-flow-and-functions.md) - focus on the *shape* for now):
```python runnable
temperature = 30
if temperature > 25:
    print("It's warm")
    print("Wear a t-shirt")
print("Done checking")
```
*What just happened:* The line ending in `:` opens a block. The two **indented** lines run only when
`temperature > 25`; the last `print`, back at the left margin, is outside the block and always runs.
Since 30 > 25, all three lines print:
```console
It's warm
Wear a t-shirt
Done checking
```

📝 **Terminology.** The `:` and the indented lines under it form a **block** (also called a *suite*).
Standard is **4 spaces** per level - don't mix tabs and spaces, and let your editor insert four on Tab.

⚠️ **`IndentationError` and `TabError`.** Because spacing *is* the structure, getting it wrong is a real
error, not a style nitpick. Indent a line that shouldn't be, or mix tabs with spaces, and Python refuses
to run:
```console
  File "warm.py", line 3
    print("It's warm")
    ^
IndentationError: unexpected indent
```
This confuses everyone at first. Fix: make every line in the block start at the same column, spaces
only - set your editor to "insert spaces for tabs" and it'll handle this for you.

💡 **Key point.** You don't indent because it looks nice - the indentation *is* how you tell Python which
lines belong together. Read it the way you'd read braces in other languages.

## Variables - names pointing at values

**What it actually is.** A variable is a *name* attached to a value so you can refer to it later. Create
one with `=` - read it as "let this name refer to this value," not as math equality.
```python runnable
name = "Ada"
age = 36
print(name)
print(age)
```
*What just happened:* `name = "Ada"` made `name` refer to the text `"Ada"`; `age = 36` made `age` refer
to `36`. Each `print` looked up what the name points at and showed it:
```console
Ada
36
```

Re-point a name at a new value any time - that's the whole idea of a *variable*:
```python runnable
score = 10
score = score + 5
print(score)
```
*What just happened:* `score + 5` was computed first using the *current* value of `score` (10), giving
15; then `score =` re-pointed the name at that value, so it prints `15`. The name didn't "change" - you
pointed it somewhere new.

## The core types

Every value in Python has a **type** - what kind of thing it is. You'll use five constantly. Ask any
value its type with the built-in `type()`:
```python runnable
print(type(36))
print(type(3.14))
print(type("hello"))
print(type(True))
print(type(None))
```
*What just happened:* `type()` reports each value's type:
```console
<class 'int'>
<class 'float'>
<class 'str'>
<class 'bool'>
<class 'NoneType'>
```

What each is *for*:

- **`int`** - a whole number, no decimal point: `36`, `0`, `-7`. No size limit; grows as large as memory
  allows.
- **`float`** - a number *with* a decimal point: `3.14`, `2.0`, `-0.5`. Short for *floating-point*, how
  computers store fractional numbers.
- **`str`** - a **string**: text in quotes. `"hello"`, `'Ada'`. Single or double both work; pick one and
  be consistent.
- **`bool`** - a **boolean**: `True` or `False` (capitalized - Python is picky). The type of any yes/no
  answer, like a comparison's result.
- **`None`** - a special value meaning "nothing here / no value yet." Its type is `NoneType`; there's
  only ever one `None`. You'll meet it as the default result of functions that don't return anything.

📝 **Terminology.** A **string** is a sequence of characters - text. The quotes aren't part of the
value; they just tell Python "the text starts here and ends there."

## Dynamic typing - names don't have a fixed type

**What it actually is.** Some languages require declaring "this variable holds an integer," and it can
*only* ever hold integers. Python doesn't work that way: a name can point at one type now and a
different type later - the *value* has a type, the *name* is just a label.
```python runnable
x = 42
print(type(x))
x = "now I'm text"
print(type(x))
```
*What just happened:* `x` first pointed at an `int`, then got re-pointed at a `str`. Python allows this
freely, so it prints two different types:
```console
<class 'int'>
<class 'str'>
```
This is called **dynamic typing**: flexible and quick to write, but nothing stops you from putting the
wrong kind of value in a name - you only find out when something breaks later. Keep each variable
holding one *kind* of thing to avoid most of that pain.

## f-strings - putting values into text

You'll constantly build strings out of fixed text plus values. The clean, modern way is an **f-string**:
a string prefixed with `f`, where anything inside `{ }` is replaced by that expression's value.
```python runnable
name = "Ada"
age = 36
print(f"{name} is {age} years old")
```
*What just happened:* The `f` before the quote marks an f-string. Python evaluated `{name}` and `{age}`
and dropped their values straight into the text:
```console
Ada is 36 years old
```
Any expression can go inside the braces, not only a bare name:
```python runnable
price = 4
print(f"Two coffees cost {price * 2} dollars")
```
*What just happened:* Python computed `price * 2` (8) and slotted the result in:
```console
Two coffees cost 8 dollars
```

💡 **Key point.** Reach for f-strings whenever you mix text and values - readable, fast, modern. (Older
code uses `"%s" % x` or `.format()`; you'll see those around but don't need to write them.)

## Two number traps that bite beginners

⚠️ **`=` vs `==` - assignment vs comparison.** A single `=` *assigns* (points a name at a value); a
double `==` *compares* two values and returns a boolean. Mixing them up is a common early mistake:
```python runnable
x = 5
print(x == 5)
print(x == 6)
```
*What just happened:* `x = 5` assigned. `x == 5` asked "is x equal to 5?" - `True` - and `x == 6` asked
the same about 6 - `False`:
```console
True
False
```
Say it in your head as you type: `=` is "set to," `==` is "is equal to?"

⚠️ **Integer vs float division - `/` always gives a float.** `/` *always* produces a `float`, even when
the numbers divide evenly. For whole-number division, use `//`:
```python runnable
print(7 / 2)
print(8 / 2)
print(7 // 2)
print(7 % 2)
```
*What just happened:* `/` gave floats (note `4.0`, not `4`). `//` did *floor division* - divide, discard
the remainder, giving a whole number. `%` (*modulo*) gave the **remainder**:
```console
3.5
4.0
3
1
```
This surprises people coming from languages where `/` on two integers stays an integer - in Python, `/`
always means a decimal. Use `//` for the whole part, `%` for what's left over (handy for "is this number
even?" - `n % 2 == 0`).

## Recap

1. **Indentation is structure.** A `:` opens a block; indented lines under it belong to it. Use 4
   spaces, never tabs - getting it wrong is a real `IndentationError`.
2. A **variable** is a name pointing at a value; `=` means "let this name refer to," re-pointable any
   time.
3. Five everyday types: **`int`** (whole), **`float`** (decimal), **`str`** (text), **`bool`**
   (`True`/`False`), **`None`** (no value). `type(x)` tells you which.
4. Python is **dynamically typed** - a name can hold and change type. Flexible, but keep each name to
   one *kind* of thing.
5. **f-strings** (`f"{name} is {age}"`) slot values into text; reach for them by default.
6. `=` assigns, `==` compares. `/` always yields a `float`; use `//` for whole-number division, `%` for
   the remainder.

Next: the lists, tuples, dicts, and sets you'll use to hold collections of things.


---

# Collections - Lists, Tuples, Dicts & Sets

A single value only gets you so far. Real programs juggle *groups* of things: a list of users, a price
per product, a set of tags. Python's four built-in collection types each have a job - reach for the
wrong one and your code fights you. Let's meet all four, then the trap that catches everyone.

## List - an ordered, changeable sequence

**What it actually is.** A **list** is an ordered row of values you can add to, remove from, or
rearrange - your default "bunch of things in order." Write one with square brackets `[]`.
```python runnable
fruits = ["apple", "banana", "cherry"]
print(fruits)
print(len(fruits))
```
*What just happened:* A list of three strings. `len()` reports how many items it holds:
```console
['apple', 'banana', 'cherry']
3
```

**Indexing** - reach in by position. Python counts from **0**; negative numbers count from the end:
```python runnable
fruits = ["apple", "banana", "cherry"]
print(fruits[0])
print(fruits[-1])
```
*What just happened:* `fruits[0]` is the *first* item (position zero, not one - this catches everyone
early); `fruits[-1]` is the *last*, counting backward:
```console
apple
cherry
```

⚠️ **Counting starts at 0.** A 3-item list has positions `0`, `1`, `2`. `fruits[3]` runs off the end and
raises `IndexError: list index out of range` - the last valid position is always `len - 1`.

**Changing it** - a list is *mutable* (changeable), so you can add and modify in place:
```python runnable
fruits = ["apple", "banana"]
fruits.append("cherry")
fruits[0] = "apricot"
print(fruits)
```
*What just happened:* `.append()` added an item to the end; `fruits[0] = "apricot"` replaced the first:
```console
['apricot', 'banana', 'cherry']
```

📝 **Terminology.** **Mutable** means "can be changed after it's created." **Immutable** means "fixed
once created." This distinction governs how every collection behaves - keep it in mind.

## Tuple - an ordered, *fixed* sequence

**What it actually is.** A **tuple** is like a list, but **immutable** - once made, you can't add,
remove, or change items. Write one with parentheses `()`, for a group of values that *belongs together
and shouldn't change*: coordinates, a row from a database, an RGB color.
```python runnable
point = (4, 5)
print(point[0])
print(point[1])
```
*What just happened:* A tuple of two numbers, read by index, exactly like a list:
```console
4
5
```

The difference shows up the moment you try to change one:
```python
point = (4, 5)
point[0] = 9
```
*What just happened:* Python refuses, because tuples are immutable:
```console
Traceback (most recent call last):
  File "point.py", line 2, in <module>
    point[0] = 9
    ~~~~~^^^
TypeError: 'tuple' object does not support item assignment
```
That "can't change me" guarantee is the *point* - it tells anyone reading the code (and Python itself)
these values are fixed.

💡 **Key point.** List vs tuple is about intent. **List**: expected to grow, shrink, or reorder.
**Tuple**: a fixed group of related values that travels together and won't change.

## Dict - lookups by key

**What it actually is.** A **dictionary** (`dict`) stores **key → value** pairs, looked up by a
meaningful *key* instead of position - the tool for "given X, what's its Y?": given a username, their
age; given a product, its price. Write one with curly braces and `key: value` pairs.
```python runnable
ages = {"ada": 36, "linus": 54}
print(ages["ada"])
print("ada" in ages)
```
*What just happened:* `ages["ada"]` looked up the value under key `"ada"`; `in` asks whether a key
exists, giving a boolean:
```console
36
True
```

Adding and updating use the same square-bracket syntax:
```python runnable
ages = {"ada": 36}
ages["grace"] = 85
ages["ada"] = 37
print(ages)
```
*What just happened:* Assigning to a *new* key added a pair; assigning to an *existing* one updated it:
```console
{'ada': 37, 'grace': 85}
```

⚠️ **`KeyError` - asking for a key that isn't there.** Looking up a missing key with `[]` doesn't return
`None` - it crashes:
```console
Traceback (most recent call last):
  File "ages.py", line 2, in <module>
    print(ages["bob"])
          ~~~~^^^^^^^
KeyError: 'bob'
```
When unsure a key exists, use `.get()`, which returns `None` (or a chosen default) instead of crashing:
```python runnable
ages = {"ada": 36}
print(ages.get("bob"))
print(ages.get("bob", 0))
```
*What just happened:* `.get("bob")` returned `None` for the missing key; `.get("bob", 0)` returned your
supplied default `0`:
```console
None
0
```

## Set - a bag of unique items

**What it actually is.** A **set** holds *unique* items with no duplicates and no particular order -
reach for it for "what distinct things are here?" or "is this present?": membership and uniqueness, not
position. Write one with curly braces (just values, no `key: value`).
```python runnable
seen = {1, 2, 2, 3, 3, 3}
print(seen)
print(2 in seen)
```
*What just happened:* Duplicates collapsed automatically - a set keeps only one of each. `in` checks
membership:
```console
{1, 2, 3}
True
```
A favorite real use: strip duplicates from a list by passing it through a set.
```python runnable
tags = ["python", "web", "python", "api", "web"]
unique = set(tags)
print(unique)
```
*What just happened:* `set(tags)` built a set from the list, discarding repeats. Order isn't guaranteed,
so yours may print differently:
```console
{'python', 'web', 'api'}
```

## Slicing - grab a *range* of a sequence

For lists, tuples, and strings, pull out a *slice* - a sub-range - with `[start:stop]`. `start` is
included; `stop` is **not**.
```python runnable
nums = [10, 20, 30, 40, 50]
print(nums[1:3])
print(nums[:2])
print(nums[2:])
```
*What just happened:* `nums[1:3]` took positions 1 and 2 - *up to but not including* 3. Omitting `start`
means "from the beginning"; omitting `stop` means "to the end":
```console
[20, 30]
[10, 20]
[30, 40, 50]
```
The same works on strings, since a string is also a sequence:
```python runnable
word = "Python"
print(word[0:3])
```
*What just happened:* Took characters at positions 0, 1, 2 - again, stopping *before* 3:
```console
Pyt
```

📝 **Terminology.** "Stop is exclusive" means the `stop` index is the first one *left out* - `nums[1:3]`
gives two items, not three. Feels odd at first, becomes second nature.

## The trap: aliasing - two names, one list

This produces the most baffling beginner bugs, so meet it on purpose.

**What's really going on.** Write `b = a` where `a` is a list, and you do **not** get a copy - both
names point at the *exact same list in memory*. Change it through one name and the other shows the
change too, because there's only one list.
```python runnable
a = [1, 2, 3]
b = a
b.append(4)
print(a)
print(b)
```
*What just happened:* `b = a` made `b` a second name for the *same* list. `b.append(4)` changed that
shared list, so `a` shows the `4` too:
```console
[1, 2, 3, 4]
[1, 2, 3, 4]
```
For a separate, independent copy, ask for one explicitly with `.copy()` (or `list(a)`):
```python runnable
a = [1, 2, 3]
b = a.copy()
b.append(4)
print(a)
print(b)
```
*What just happened:* `.copy()` made a brand-new list with the same contents - `a` and `b` are now
independent, so appending to `b` leaves `a` untouched:
```console
[1, 2, 3]
[1, 2, 3, 4]
```

⚠️ **This only bites mutable collections.** Lists, dicts, and sets are mutable, so aliasing matters.
Numbers, strings, and tuples are immutable - you can't change them in place, so sharing one is harmless.
The rule: **assignment never copies; it makes another name for the same object.**

## Recap

1. **List** `[]` - ordered and *changeable*; your default sequence. Index from `0`, `-1` is the last.
2. **Tuple** `()` - ordered but *fixed* (immutable); for groups of values that shouldn't change.
3. **Dict** `{key: value}` - look up values by key; use `.get()` to avoid `KeyError` on missing keys.
4. **Set** `{a, b, c}` - unique items, no order; great for deduping and membership tests.
5. **Slicing** `[start:stop]` grabs a sub-range; `stop` is *excluded*.
6. **Aliasing:** `b = a` makes two names for *one* list, not a copy. Use `.copy()` for an independent
   one - assignment never copies.

Next: making programs *decide* and *repeat* with `if`/`else`, loops, and functions - plus the famous
mutable-default-argument trap.


---

# Control Flow & Functions

So far your programs run straight down, every line once. Real programs *make choices* ("if logged in,
show the dashboard") and *repeat work* ("for each order, send an email"), and once you've written useful
logic, you want to *name it and reuse it* instead of copy-pasting. Deciding, repeating, and packaging -
control flow and functions - are where code starts to feel powerful.

Remember the indentation rule from [Phase 2](02-syntax-values-and-types.md): `:` opens a block and the
indented lines beneath belong to it. Every structure here uses it.

## `if` / `elif` / `else` - making a decision

**What it actually is.** An `if` runs a block *only when* a condition is true. Add `elif` ("else if") for
more conditions, and `else` for "none of the above." Python checks top to bottom, runs the **first**
matching block, then skips the rest.
```python runnable
score = 72
if score >= 90:
    print("A")
elif score >= 80:
    print("B")
elif score >= 70:
    print("C")
else:
    print("F")
```
*What just happened:* Python checked `score >= 90` (false), then `>= 80` (false), then `>= 70` (true),
ran that block, and stopped without reaching `else`:
```console
C
```
As a picture - Python falls through the checks until one matches:

```mermaid
flowchart TD
  start([score = 72]) --> a{score >= 90?}
  a -- yes --> A[print A]
  a -- no --> b{score >= 80?}
  b -- yes --> B[print B]
  b -- no --> c{score >= 70?}
  c -- yes --> C[print C]
  c -- no --> E[print F]
```

📝 **Terminology.** A **condition** is any expression evaluating to `True` or `False` - usually a
comparison like `>=`, `==`, `!=` (not equal), `<`, `>`. The block under `if` runs when it's `True`.

## Truthiness - what counts as "true"

**What it actually is.** Python lets you use *any* value as a condition, not only `True`/`False` - it
asks "is this *truthy* or *falsy*?" The falsy values are "empty or nothing"; almost everything else is
truthy.
```python runnable
print(bool(0), bool(""), bool([]), bool(None))
print(bool(42), bool("hi"), bool([1, 2]))
```
*What just happened:* `bool()` shows how Python judges each value as a condition: zero, the empty
string, the empty list, and `None` are all **falsy**; a nonzero number, non-empty string, and non-empty
list are **truthy**:
```console
False False False False
True True True
```
This enables natural checks - instead of `if len(items) > 0:`, write:
```python runnable
items = []
if items:
    print("There are items")
else:
    print("The list is empty")
```
*What just happened:* The empty list is falsy, so the `else` ran:
```console
The list is empty
```

💡 **Key point.** "Empty or zero or nothing" is falsy; everything else truthy. `if my_list:` reads as
"if the list has anything in it" - clean and Pythonic.

## `for` - do something for each item

**What it actually is.** A `for` loop walks a collection, running its block *once per item*, each handed
to a name you choose.
```python runnable
for fruit in ["apple", "banana", "cherry"]:
    print(fruit)
```
*What just happened:* The loop took each item in turn, pointed `fruit` at it, and ran the block - three
items, three runs:
```console
apple
banana
cherry
```
To repeat a fixed number of times, loop over `range(n)`, which produces `0` up to (but not including)
`n`:
```python runnable
for i in range(3):
    print(i)
```
*What just happened:* `range(3)` yielded `0`, `1`, `2` - stopping *before* 3, the same "stop is
exclusive" rule as slicing:
```console
0
1
2
```

## `while` - repeat until a condition turns false

**What it actually is.** A `while` loop repeats its block *as long as* a condition stays true. Reach for
it when you don't know in advance how many times you'll loop - you loop until something changes.
```python runnable
n = 3
while n > 0:
    print(n)
    n = n - 1
```
*What just happened:* The loop ran while `n > 0`, printing `n` then shrinking it by 1 each pass, until it
hit 0 and the condition went false:
```console
3
2
1
```

⚠️ **The infinite loop.** A `while` only stops when its condition becomes false, so *something inside the
loop must move it toward false*. Drop the `n = n - 1` line above and `n` stays 3 forever, printing without
end. If a program ever "hangs," suspect an infinite loop; press **Ctrl-C** to stop it.

## Functions - name a piece of logic and reuse it

**What it actually is.** A **function** is a named, reusable block of instructions: *define* it once with
`def`, then *call* it whenever needed, with different inputs each time - how you avoid copy-pasting logic.
```python runnable
def greet(name):
    return f"Hello, {name}!"

print(greet("Ada"))
print(greet("Linus"))
```
*What just happened:* `def greet(name):` defined a function taking one **parameter**, `name`; `return`
hands a value back. Each call supplied a different name, giving two different results:
```console
Hello, Ada!
Hello, Linus!
```

📝 **Terminology.** A **parameter** is the name in the definition (`name`); an **argument** is the actual
value passed when calling (`"Ada"`). `return` sends a value back out of the function.

**Defaults** let a parameter be optional by giving it a fallback value:
```python runnable
def greet(name, greeting="Hello"):
    return f"{greeting}, {name}!"

print(greet("Ada"))
print(greet("Ada", "Hi"))
```
*What just happened:* Without a `greeting` argument, it falls back to `"Hello"`; supply one and yours
wins:
```console
Hello, Ada!
Hi, Ada!
```

**`return` vs printing - a crucial difference.** A function that *prints* shows text but hands back
nothing usable; one that *returns* gives a value you can store and work with. No `return` means it hands
back `None`:
```python runnable
def show(x):
    print(x)

result = show(5)
print(result)
```
*What just happened:* `show(5)` printed `5`, but with no `return`, the call evaluated to `None` - stored
in `result` and printed on the second line:
```console
5
None
```
To *use* a function's output later, it must `return` it, not just `print` it - printing is for humans,
returning is for feeding the rest of your program.

## The classic trap: mutable default arguments

This one bites *experienced* developers too, so it's worth meeting head-on.

**What goes wrong.** Give a parameter a **mutable** default (like a list), and it's created **once**,
when the function is defined - then *shared across every call*, never fresh, which is almost never what
you want.
```python runnable
def add_item(item, basket=[]):
    basket.append(item)
    return basket

print(add_item("apple"))
print(add_item("banana"))
```
*What just happened:* You'd expect each call to start with an empty basket. Instead both share the
*same* default list, so the second call sees the first call's leftovers:
```console
['apple']
['apple', 'banana']
```
That `['apple', 'banana']` on the second line is the bug - `"apple"` shouldn't be there. The shared
default silently accumulates across calls.

**The fix** - a pattern to adopt every time: default to `None`, then create a fresh value *inside* the
function.
```python runnable
def add_item(item, basket=None):
    if basket is None:
        basket = []
    basket.append(item)
    return basket

print(add_item("apple"))
print(add_item("banana"))
```
*What just happened:* Now each call with no basket gets a brand-new list, so they don't bleed together:
```console
['apple']
['banana']
```

⚠️ **Never use a mutable default (`[]`, `{}`, `set()`) directly.** Default to `None` and build the real
value inside the function. Memorize this - it's a genuine sharp edge, coming straight from the aliasing
idea in [Phase 3](03-collections.md): the default list is one shared object.

## Recap

1. **`if` / `elif` / `else`** runs the *first* matching block. Conditions are expressions evaluating to
   `True`/`False`.
2. **Truthiness:** empty/zero/`None` are falsy, everything else truthy - so `if my_list:` means "if it
   has items."
3. **`for`** loops once per item (use `range(n)` for a count); **`while`** loops until its condition goes
   false - make sure something moves it there.
4. **`def`** defines a function; **parameters** name its inputs, **`return`** hands a value back.
   **Defaults** make parameters optional.
5. `return` gives a usable value; a function with no `return` yields `None`.
6. **Never use a mutable default argument.** Default to `None` and build the list/dict inside the
   function.

Next: importing code, the standard library, writing your own modules, and laying out a real project.


---

# Modules & Project Layout

One file is fine for a script, but once a program grows past a screen or two, cramming everything into
`main.py` turns it into a haystack. Python's answer is the **module**: every `.py` file is one, and
`import` pulls code from one file into another. That same mechanism gives you the **standard library** -
a huge toolbox that ships with Python.

## A module is just a `.py` file

**What it actually is.** A **module** is a single Python file. Its functions, variables, and (later)
classes can be *imported* - borrowed - into another file. Nothing special to do to create one; writing
`greetings.py` already makes a module named `greetings`.

A tiny two-file program. First, the module:
```python
# greetings.py
def shout(text):
    return text.upper() + "!"

PI = 3.14159
```
*What just happened:* One function, one variable, nothing visible on its own - a toolbox waiting to be
opened by another file. (The `#` starts a **comment**, a note for humans that Python ignores.)

## `import` - borrow code from another module

A second file that uses it. Two common styles, differing in *how you then refer to the borrowed names*:
```python
# main.py
import greetings
from greetings import shout

print(greetings.PI)
print(shout("hello"))
```
*What just happened:* `import greetings` brought in the whole module - reach into it with a dot, like
`greetings.PI`. `from greetings import shout` pulled out *just* that name, callable directly without a
prefix:
```console
3.14159
HELLO!
```

📝 **Terminology.** `import module` brings in the module as a namespace (use `module.thing`); `from
module import thing` brings a specific name straight in (use `thing`). Both run the imported file once -
the difference is only how you spell the names afterward.

⚠️ **`ModuleNotFoundError`.** If Python can't find what you're importing, you get:
```console
ModuleNotFoundError: No module named 'greetings'
```
Two usual causes: the file isn't in the folder you're running from (Python looks alongside the file you
ran), or the name is misspelled. You import `greetings`, **not** `greetings.py` - drop the extension.

## The standard library - batteries included

**What it actually is.** Python ships with a large collection of ready-made modules, the **standard
library** - math, dates, randomness, file paths, JSON, and more. Import them like your own; nothing to
install, they're already there.
```python runnable
import math
from random import randint

print(math.sqrt(16))
print(randint(1, 6))
```
*What just happened:* `math.sqrt(16)` returned the square root as a float; `randint(1, 6)` returned a
random whole number from 1 to 6 (a die roll) - yours will vary:
```console
4.0
3
```

💡 **Key point.** "Is there already a module for this?" is the right first question in Python. The
standard library covers an enormous amount, and the wider ecosystem covers most of the rest - that's
[Phase 8](08-ecosystem-and-tooling.md). Reach for existing, tested code before writing your own.

## `if __name__ == "__main__"` - run-directly vs imported

This line shows up in nearly every Python file and looks cryptic until you see the problem it solves.

**The problem.** When you `import` a file, Python *runs the whole file* to define its functions and
variables. Fine for `def`s - but any *top-level* code (a `print`, a call) runs too, the moment someone
imports it. You usually don't want a module's demo code firing just because another file borrowed one
function from it.

**The mechanism.** Python sets a built-in variable, `__name__`, differently depending on how the file is
used: `"__main__"` when you **run it directly**, the *module's own name* when it's **imported**. So you
guard "run this only when executed directly" behind a check on it.
```python runnable
# greetings.py
def shout(text):
    return text.upper() + "!"

if __name__ == "__main__":
    print(shout("running directly"))
```
*What just happened:* The `def` always defines the function, whether imported or run directly. The
guarded block runs *only* when this file is the one you launched:
```console
$ python3 greetings.py
RUNNING DIRECTLY!
```
But `import greetings` from another file defines `shout` and *skips* the guarded block, because
`__name__` is `"greetings"`, not `"__main__"` - importing it stays silent, exactly what you want.

📝 **Terminology.** `__name__` is a variable Python sets for you: `"__main__"` ⇒ "this file was run
directly"; the module name ⇒ "this file was imported." The `if __name__ == "__main__":` block is
conventionally where a script's *starting point* lives.

## A sane small project layout

As a program grows, group related code into files in sensible places. A clean, common starting shape:

```mermaid
flowchart TD
  root[my_project/] --> main[main.py - the entry point]
  root --> pkg[app/ - your package]
  root --> reqs[requirements.txt - dependencies]
  root --> readme[README.md - what & how to run]
  pkg --> init[__init__.py - marks app/ as a package]
  pkg --> greet[greetings.py - a module]
  pkg --> models[models.py - a module]
```

What each piece is for:

- **`main.py`** - the entry point you run (`python3 main.py`); imports from your package and kicks
  things off inside an `if __name__ == "__main__":` block.
- **`app/`** - a folder of your real code, split into focused modules (`greetings.py`, `models.py`, …).
  A folder of modules like this is a **package**.
- **`app/__init__.py`** - an (often empty) file whose presence tells Python "this folder is a package."
  Import from it as `from app.greetings import shout`.
- **`requirements.txt`** - a list of outside libraries your project needs (covered in
  [Phase 8](08-ecosystem-and-tooling.md)).
- **`README.md`** - a plain-text note on what the project is and how to run it.

📝 **Terminology.** A **module** is one `.py` file; a **package** is a folder of modules (marked by
`__init__.py`). "Import from the `app` package" means reach into that folder's files.

⚠️ **Don't over-organize on day one.** A 30-line script doesn't need a package - a single file is
correct. Add structure when the file gets unwieldy, not before. Start flat; split when it hurts.

## Recap

1. Every `.py` file is a **module**; `import` borrows its code into another file.
2. `import module` ⇒ use `module.thing`; `from module import thing` ⇒ use `thing` directly. Import by
   name, without `.py`.
3. The **standard library** ships with Python - `math`, `random`, and many more - no install needed.
   Check it before writing your own.
4. `if __name__ == "__main__":` runs a block *only when the file is executed directly*, not when it's
   imported. It's where a script's starting point goes.
5. A small project: a `main.py` entry point, an `app/` **package** (folder of modules with
   `__init__.py`), plus `requirements.txt` and a `README.md`. Add structure only when you need it.

Next: modeling *things* - objects and the classes that define them - so data and its operations live
together.


---

# Objects & Classes - Python's OOP

You've been using objects since Phase 2 without anyone calling them that. A string has a `.upper()`
method; a list has `.append()`. Each is an object: *data* (the characters, the items) glued to the
*behavior* that works on it. Classes are how you make your own.

"OOP" scares people because it arrives wrapped in vocabulary - encapsulation, polymorphism, abstraction.
Ignore that for now: exactly one idea sits underneath it, and once it clicks, the keywords stop being
spells and start being obvious.

## The one idea: bundle data with the behavior that acts on it

**What it actually is.** A **class** is a blueprint saying "things of this kind hold *this* data and can
do *these* things." An **object** (or **instance**) is one actual thing built from that blueprint - the
class is the cookie cutter, the objects are the cookies.

📝 **Class** - the template. **Instance / object** - one concrete thing made from it. Write the class
once and stamp out as many instances as you like, each with its own copy of the data.

**Why this exists.** Imagine modeling a dog without classes: a loose `name` string, an `age` number,
separate functions `bark(name)` and `birthday(age)` floating elsewhere. Nothing keeps them together -
keeping them in sync is on you. OOP's answer: one box for the data and its functions.

> 💡 **Key point.** Everything in this phase is one sentence repeated: *the data and the behavior that
> belongs with it live together in an object.* Lost? Return to that line.

## Your first class

**A real example.** A `Dog` - read the comments, they carry the whole lesson.

```python runnable
class Dog:
    def __init__(self, name, age):   # the constructor: runs when you make a Dog
        self.name = name             # store data ON this particular dog
        self.age = age

    def bark(self):                  # a method: behavior that belongs to a Dog
        return f"{self.name} says woof!"

rex = Dog("Rex", 3)                  # build one instance
print(rex.bark())
print(rex.age)
```
```console
$ python dogs.py
Rex says woof!
3
```
*What just happened:* `Dog("Rex", 3)` ran `__init__` with `name="Rex"` and `age=3`, storing those on the
new dog as `self.name` and `self.age`. `rex.bark()` then read its own `self.name` back out.

### `__init__` - the setup ritual

**What it actually is.** `__init__` (two underscores each side, said "dunder init") is the
**constructor**: a special method Python runs automatically when you create an instance, to set up its
starting data.

📝 **Dunder** - short for "double underscore." Names like `__init__` are Python's hooks: you define them
and Python calls them at the right moment. You almost never call `__init__` yourself - `Dog(...)` calls
it for you.

### `self` - the "this particular object" handle

**What it actually is.** `self` is the first parameter of every method, referring to *the specific
instance called*. Write `rex.bark()`, and Python quietly passes `rex` in as `self` - so inside `bark`,
`self.name` means "Rex's name," not some other dog's.

⚠️ **Gotcha - forgetting `self`.** Every method's first parameter must be `self`, and every reference to
the object's own data must go through it. Write `def bark():` (no `self`) or `return f"{name} says woof!"`
(no `self.`) and Python throws a `TypeError` or `NameError`. This trips up everyone coming from languages
where `this` is implicit - in Python it's explicit and always spelled out.

```console
$ python dogs.py
TypeError: Dog.bark() takes 0 positional arguments but 1 was given
```
*What just happened:* `def bark():` had no parameter, but `rex.bark()` still passes `rex` in as the first
argument - an argument the method wasn't expecting. Fix: `def bark(self):`.

### Attributes vs. methods

📝 **Attribute** - data stored on an object (`rex.name`, `rex.age`). **Method** - a function defined in
the class that acts on the object (`rex.bark()`). Read an attribute with no parentheses; call a method
with them.

## Inheritance - describe only what's different

**The problem it solves.** Suppose you now want a `Cat` and a `Cow` too. All have a name and age; all eat
and sleep. Only their sound differs. Copy-pasting the shared parts into three classes means three places
to fix when the logic changes - and they *will* drift apart.

**What it actually is.** **Inheritance** lets one class build on another: the child gets everything the
parent has, and you add or override only what's different. Pull the common parts into an `Animal` parent
and let `Dog` inherit from it.

```python runnable
class Animal:
    def __init__(self, name):
        self.name = name

    def speak(self):
        return f"{self.name} makes a sound."

class Dog(Animal):               # Dog IS an Animal, plus a little more
    def speak(self):             # override: dogs are more specific
        return f"{self.name} says woof!"

generic = Animal("Thing")
rex = Dog("Rex")
print(generic.speak())
print(rex.speak())
```
```console
$ python animals.py
Thing makes a sound.
Rex says woof!
```
*What just happened:* `Dog(Animal)` means "a Dog is an Animal." `Dog` never defined `__init__`, so it
inherited `Animal`'s - why `Dog("Rex")` still sets `self.name`. But `Dog` did define its own `speak`, so
that version wins. This is **overriding**: same method name, more specific behavior.

📝 **Parent / child** (also **superclass / subclass**) - `Animal` is the parent, `Dog` the child.
**Override** - a child redefining an inherited method to specialize it.

That relationship as a picture:

```mermaid
classDiagram
  Animal <|-- Dog
  Animal <|-- Cat
```

*One idea:* `Dog` and `Cat` both *are* `Animal`s (arrows point up to the parent). They share `name` and
the idea of `speak()`, but each gives it its own voice.

### `super()` - reuse the parent's work

When a child needs *extra* setup but still wants the parent's, `super()` calls the parent's version so
you don't repeat it.

```python
class Dog(Animal):
    def __init__(self, name, breed):
        super().__init__(name)   # let Animal set up self.name
        self.breed = breed       # then add what's new to Dog

rex = Dog("Rex", "Beagle")
print(rex.name, "-", rex.breed)
```
```console
$ python super_demo.py
Rex - Beagle
```
*What just happened:* `super().__init__(name)` ran `Animal.__init__`, setting `self.name`; then `Dog`
added its own `self.breed`. The child reused the parent's setup instead of copy-pasting `self.name = name`.

⚠️ **Gotcha - inheritance overuse.** People reach for inheritance too eagerly. Deep chains (class extends
class extends class, four levels down) get impossible to follow, since understanding one object means
mentally merging four files. Use it only for genuine, permanent "is-a" relationships, kept shallow. When
two things merely *share some parts*, prefer giving one object the other as a field
(`self.engine = Engine()`). The full trade-off - inheritance vs. composition, and the other paradigm
entirely - is laid out in [OOP vs. Functional](/guides/oop-vs-functional).

**Why this saves you later.** Shared logic changes once in the parent and every child updates for free.
Add a `Cat` tomorrow and you describe only its sound, not a whole animal from scratch.

## Recap

1. A **class** is a blueprint; an **object / instance** is one thing built from it, with data and
   behavior living together inside.
2. **`__init__`** runs automatically when you create an instance, setting up its starting data.
3. **`self`** is "this particular object" - the first parameter of every method, and how a method reads
   its own data. Forget it and you'll get a `TypeError`.
4. **Attributes** are data (`rex.name`); **methods** are behavior (`rex.bark()`).
5. **Inheritance** (`class Dog(Animal)`) lets a child reuse a parent and override only what differs;
   **`super()`** calls the parent's version. Use it sparingly, for true "is-a" relationships.

OOP is one of two big ways to organize code; the other, functional, keeps functions and data apart on
purpose. Next: what to do when things go wrong, and how to read and write files.


---

# Errors & I/O - Exceptions and Files

Two things happen in every real program: things go wrong, and data comes from or goes to the outside
world - usually a file. Python ties these together, since reading a file is one of the most common
places things *do* go wrong (the file's missing, the disk is full, the data's garbage).

The mental shift is small but important: a beginner sees an error and thinks "my program crashed"; a
Python programmer thinks "an *exception* was raised, and I get to decide what happens next." Errors
aren't the end of the story - they're a signal you can catch.

## What an exception actually is

**What it actually is.** An **exception** is Python's way of stopping code dead and shouting "I can't do
what you asked." Divide by zero, open a missing file, or index past a list's end, and Python *raises* an
exception. Uncaught, it travels up and crashes the program, printing a traceback.

📝 **Raise** - to trigger an exception. **Traceback** - the "where it happened" lines Python prints when
an uncaught exception crashes the program. (Reading those is a skill of its own - see
[What an Error Message Tells You](/guides/what-an-error-message-tells-you).)

```python runnable
print(10 / 0)
```
```console
$ python boom.py
Traceback (most recent call last):
  File "boom.py", line 1, in <module>
    print(10 / 0)
          ~~~^~~
ZeroDivisionError: division by zero
```
*What just happened:* Dividing by zero is undefined, so Python raised a `ZeroDivisionError`. Nobody
caught it, so it bubbled up and crashed the program, printing where it happened and why. The *last*
line - `ZeroDivisionError: division by zero` - is the actual problem.

## try / except - catch what you can handle

**What it actually is.** `try` marks a block "this might fail"; `except` says "if *this kind* of failure
happens, do this instead of crashing." Catch only the specific failures you know how to handle.

```python runnable
def safe_divide(a, b):
    try:
        return a / b
    except ZeroDivisionError:
        return "can't divide by zero"

print(safe_divide(10, 2))
print(safe_divide(10, 0))
```
```console
$ python divide.py
5.0
can't divide by zero
```
*What just happened:* The first call ran the `try` block cleanly and returned `5.0`. The second raised
`ZeroDivisionError` inside the `try`, so Python jumped straight to the matching `except` and returned the
friendly message instead of crashing.

⚠️ **Gotcha - never write a bare `except:`.** It's tempting to write `except:` with no error type to
"catch everything." Don't - it swallows *every* exception, including `KeyboardInterrupt` (Ctrl-C) and
genuine bugs like a typo'd variable name, hiding them behind whatever you do next. You'll spend an
afternoon wondering why your program ignores Ctrl-C and silently misbehaves. **Always name the
exceptions you actually expect:**

```python
# BAD - hides every error, including bugs and Ctrl-C
try:
    risky()
except:
    pass

# GOOD - catch only what you understand
try:
    risky()
except (ValueError, KeyError) as err:
    print(f"handling: {err}")
```
*What just happened:* The good version catches `ValueError` and `KeyError` - the anticipated failures -
and binds the exception object to `err` for inspection. Anything else still crashes loudly, exactly what
you want for unforeseen bugs.

## finally - code that runs no matter what

**What it actually is.** A `finally` block runs whether the `try` succeeded, failed, or raised something
you didn't catch - for cleanup that *must* happen, like closing a connection or releasing a lock.

```python
def read_first_line(path):
    f = open(path)
    try:
        return f.readline()
    finally:
        f.close()        # runs even if readline() blows up
        print("file closed")
```
*What just happened:* No matter what `readline()` does - return a line or raise mid-read - `f.close()`
runs before the function returns or the error propagates. `finally` guarantees the file won't be left
open. (`with`, next, does this automatically - but it's worth seeing the manual version once.)

## raise - throw your own exception

**What it actually is.** You don't only *catch* exceptions; you `raise` them when your code hits a
situation it can't accept. A clear error raised early beats a nonsense value that explodes three
functions later.

```python runnable
def withdraw(balance, amount):
    if amount > balance:
        raise ValueError(f"can't withdraw {amount} from {balance}")
    return balance - amount

print(withdraw(100, 30))
print(withdraw(100, 500))
```
```console
$ python bank.py
70
Traceback (most recent call last):
  ...
ValueError: can't withdraw 500 from 100
```
*What just happened:* The first call was fine. The second hit the guard and *raised* a `ValueError`
saying exactly what went wrong. The caller can `try/except` it, or the crash points straight at the real
problem instead of a mysterious negative balance downstream.

## Reading and writing files with `with open(...)`

**What it actually is.** `open(path)` hands you a file object. `with` is a **context manager**: it
guarantees the file is closed when the block ends, even if an exception fires inside it - the
`finally`-cleanup from earlier, done for you.

📝 **Context manager** - anything used with `with`. It sets something up on the way in and tears it down
on the way out, automatically. Files are the classic example.

**Writing:**
```python
with open("notes.txt", "w") as f:    # "w" = write (creates/overwrites)
    f.write("first line\n")
    f.write("second line\n")
# file is automatically closed here
```
*What just happened:* `"w"` opened `notes.txt` for writing, truncating it if it already existed. We wrote
two lines (`\n` is the newline - `write` doesn't add one for you). When the `with` block ended, Python
flushed and closed the file - no `f.close()` needed.

**Reading it back:**
```python
with open("notes.txt") as f:         # no mode = read ("r") by default
    for line in f:                   # iterate line by line
        print(line.rstrip())         # rstrip() drops the trailing newline
```
```console
$ python files.py
first line
second line
```
*What just happened:* Opening with no mode defaults to read. Looping over a file object yields one line
at a time - memory-friendly even for huge files, since it never loads the whole thing at once. `rstrip()`
drops the `\n` that `print` would otherwise double up.

⚠️ **Gotcha - `"w"` erases the file.** Opening with `"w"` truncates the file to empty *before* you write
a single byte. To *add* to a file, use `"a"` (append); reach for `"w"` only to start fresh. This one has
eaten real data - check the mode before you run it.

## EAFP - "ask forgiveness, not permission"

**What it actually is.** Two styles exist for handling things that might fail. *Look before you leap*
(LBYL) checks first: "does this file exist? then open it." *Easier to Ask Forgiveness than Permission*
(EAFP) just tries it and catches the failure. Python culture strongly prefers **EAFP**: cleaner, and it
avoids a sneaky bug.

```python
# LBYL - check first (Python tends to avoid this)
import os
if os.path.exists("config.txt"):
    with open("config.txt") as f:
        data = f.read()
else:
    data = ""

# EAFP - just try it (the Pythonic way)
try:
    with open("config.txt") as f:
        data = f.read()
except FileNotFoundError:
    data = ""
```
*What just happened:* Both end with `data` set, but LBYL has a hidden flaw: the file could be *deleted in
the instant between* `exists()` returning `True` and `open()` running, and then it crashes anyway. EAFP
has no such gap - it attempts the real operation and handles the one failure it cares about.

> 💡 **Key point.** When something might not work, the Pythonic instinct is *try it and catch the
> specific failure* - not interrogate the world first and hope nothing changes.

## Recap

1. An **exception** is Python saying "I can't continue." Uncaught, it crashes with a traceback whose
   *last line* names the real problem.
2. **`try` / `except`** catches failures you know how to handle - name them; **never** use a bare
   `except:`.
3. **`finally`** runs no matter what - for cleanup that must happen.
4. **`raise`** throws your own exception when your code hits something it can't accept.
5. **`with open(...)`** reads and writes files and closes them automatically; mind that `"w"` overwrites
   and `"a"` appends.
6. **EAFP** - try the operation and catch the specific error - is the Pythonic style, and it dodges the
   race condition "check first" quietly has.

Next: the tooling that turns a script into a real project - package installs, virtual environments,
formatters, and tests.


---

# The Ecosystem & Tooling

You can write Python forever with nothing but the standard library, but the moment you want to talk to a
web API, parse a spreadsheet, or run a test suite, you'll reach for code other people wrote - and that's
where tooling matters as much as the language itself.

This phase is the everyday toolbox: installing packages, keeping them from turning your machine into a
junk drawer, and the three tools (formatter, linter, test runner) that separate "a script that works on
my laptop" from "a project a team can live in."

## pip - the package installer

**What it actually is.** **pip** is Python's package installer: it downloads libraries from **PyPI** (the
Python Package Index, a giant public repository) and makes them importable in your code.

📝 **Package** - a reusable library someone published (e.g. `requests` for HTTP, `rich` for pretty
terminal output). **PyPI** - the official place pip downloads them from.

```console
$ python -m pip install requests
Collecting requests
  Downloading requests-2.32.3-py3-none-any.whl (64 kB)
Installing collected packages: requests
Successfully installed requests-2.32.3
```
*What just happened:* pip fetched `requests` (and its dependencies) from PyPI and installed it where
Python can find it - now `import requests` works. `python -m pip`, not bare `pip`, guarantees you're
using the pip belonging to *this* Python, not another one lurking on your `PATH`.

## Virtual environments - one isolated box per project

**The problem it solves.** Say project A needs `requests` 2.20 and project B needs 2.32. Install packages
globally and the two fight over one shared pile of libraries - upgrading one silently breaks the other.
This is "dependency hell," exactly as fun as it sounds.

**What it actually is.** A **virtual environment** (venv) is a private, throwaway folder holding its own
copy of Python and its own packages, isolated from every other project and the system Python. Each
project gets its own box; what you install in one can't touch another.

```console
$ python -m venv .venv
$ source .venv/bin/activate        # macOS/Linux
(.venv) $ python -m pip install requests
```
On Windows the activation line is different:
```console
> python -m venv .venv
> .venv\Scripts\activate
(.venv) > python -m pip install requests
```
*What just happened:* `python -m venv .venv` created a folder containing a fresh, empty Python.
`activate` switched your shell to use it - the `(.venv)` prompt prefix is your sign you're inside the
box. Now `pip install` drops packages *into `.venv` only*; delete the folder and the environment is
gone, harming nothing else. To leave, run `deactivate`.

> 💡 **Key point.** Make a virtual environment for *every* project, the moment you start it - the single
> habit that prevents the most common, most baffling "it worked yesterday" failures. The `.venv` folder
> is disposable - never commit it to Git.

⚠️ **Gotcha - installing into the wrong place.** Forget to activate before `pip install`, and the package
lands in your global Python (or fails with a permissions error) - your project won't see it. Check the
`(.venv)` prefix: no prefix, no isolation.

## Recording dependencies - requirements.txt and pyproject.toml

**The problem it solves.** Your `.venv` lives only on your machine, but a teammate (or server, or
future-you on a new laptop) needs to recreate the *same* set of packages - so you record them in a file.

**`requirements.txt`** - the simple, classic format: one package per line, usually with a pinned version.

```console
$ python -m pip freeze > requirements.txt
$ cat requirements.txt
certifi==2024.7.4
charset-normalizer==3.3.2
idna==3.7
requests==2.32.3
urllib3==2.2.2
```
*What just happened:* `pip freeze` printed every installed package with its exact version, and `>` saved
that list to `requirements.txt`. Anyone can recreate your environment with
`pip install -r requirements.txt` and get identical versions - no guessing, no drift.

**`pyproject.toml`** - the modern, richer format: a single config file, defined by a Python standard,
that describes your project *and* its dependencies (and configures your tools, below). A minimal one:

```toml
[project]
name = "my-app"
version = "0.1.0"
dependencies = [
    "requests>=2.32",
]
```
*What just happened:* This declares the project's name, version, and its dependency on `requests` 2.32+.
Modern tooling reads this one file instead of scattering config across many. `requirements.txt` is fine
for a first project; `pyproject.toml` is where things head as it grows.

📝 **Pinning** - recording an *exact* version (`requests==2.32.3`) so installs are reproducible; `>=2.32`
is a looser *range* that picks up newer compatible releases. Pin for apps you deploy; ranges are common
for libraries.

📝 **You'll increasingly see `uv`.** By 2026 many teams reach for [uv](https://docs.astral.sh/uv/), a
single fast tool that bundles the install/venv/lockfile steps above (it replaces `pip` and `venv` with
`uv add`, `uv run`, and a `uv.lock`). Learn `pip` and `venv` first - they're the universal baseline every
Python install has - then `uv` is an easy step up once these click.

## The quality tools - black, ruff, pytest at a glance

These three show up on almost every serious Python project. Install them into your venv
(`pip install black ruff pytest`).

### black - the formatter

**What it actually is.** **black** is an *opinionated* code formatter: it rewrites your file into one
consistent style - spacing, line breaks, quotes - so nobody on the team argues about formatting again. It
makes the decisions; you stop thinking about them.

```console
$ black app.py
reformatted app.py

All done! ✨ 🍰 ✨
1 file reformatted.
```
*What just happened:* black rewrote `app.py` in place to match its canonical style - messy indentation
and inconsistent quotes became uniform. You don't configure much; that's the point, its whole pitch is
"no options to bikeshed over."

### ruff - the linter

**What it actually is.** A **linter** reads your code without running it and flags likely problems:
unused imports, undefined names, suspicious patterns. **ruff** is a very fast one, now the common choice.

```console
$ ruff check app.py
app.py:1:8: F401 [*] `os` imported but unused
app.py:14:5: F821 Undefined name `reqeusts`
Found 2 errors.
```
*What just happened:* ruff caught two things *before running the code*: an unused `import os`, and a
typo'd `reqeusts` (should be `requests`) that would've crashed at runtime - instant feedback instead of a
confusing traceback later.

📝 **Formatter vs. linter** - a formatter changes how code *looks* (style); a linter warns about what
code *does* (likely bugs and smells). Complementary, not competing.

### pytest - the test runner

**What it actually is.** **pytest** runs your tests: write plain functions named `test_*` that `assert`
what should be true, and pytest finds, runs, and reports pass/fail on them. (The *why* of testing is its
own topic - see [Why Test At All](/guides/why-test-at-all).)

```python
# test_math.py
def add(a, b):
    return a + b

def test_add():
    assert add(2, 3) == 5
    assert add(-1, 1) == 0
```
```console
$ pytest
========================= test session starts =========================
collected 1 item

test_math.py .                                                  [100%]

========================== 1 passed in 0.01s ==========================
```
*What just happened:* pytest discovered `test_add` (it looks for `test_*` functions automatically), ran
it, and the `assert`s held, printing a green dot and `1 passed`. Break the math and that dot becomes an
`F` with a clear diff of expected-vs-actual. No boilerplate test class required; plain functions and
`assert` are enough.

## Recap

1. **pip** installs packages from PyPI; prefer `python -m pip` so you hit the right Python.
2. A **virtual environment** (`python -m venv .venv` + activate) gives each project its own isolated box
   of packages - make one per project, every time, and never commit `.venv`.
3. **requirements.txt** (`pip freeze`) and **pyproject.toml** record your dependencies so anyone can
   recreate the environment.
4. **black** formats, **ruff** lints, **pytest** runs tests - the everyday trio that keeps a project
   clean, correct, and pleasant to work in.

Next: the part that makes code feel *Pythonic* - the idioms locals use, and the gotchas that bite
everyone exactly once.


---

# Idioms & Common Gotchas

There's a point where Python stops feeling like "a language I'm translating my old habits into" and
starts feeling like its own thing - when you learn the **idioms**, patterns Python programmers reach for
so often that code *not* using them looks faintly foreign.

This phase has two halves: the idioms worth adopting on purpose, then a cheat-card of gotchas - small,
surprising behaviors that bite nearly every Python programmer exactly once. Read them *before* they bite
you; that's the point of having them in one place.

## The idioms worth adopting

### Comprehensions - build a list (or dict) in one expression

**What it actually is.** A **comprehension** builds a new collection by describing it, instead of
starting empty and appending in a loop - close to how you'd say it out loud.

```python runnable
nums = [1, 2, 3, 4, 5]

# the long way
squares = []
for n in nums:
    squares.append(n * n)

# the idiom
squares = [n * n for n in nums]

# with a filter
evens = [n for n in nums if n % 2 == 0]

# a dict comprehension
lengths = {word: len(word) for word in ["hi", "hello"]}
print(squares, evens, lengths)
```
```console
$ python comp.py
[1, 4, 9, 16, 25] [2, 4] {'hi': 2, 'hello': 5}
```
*What just happened:* `[n * n for n in nums]` produced the same list as the four-line loop, in one line
reading "n squared, for each n in nums." `if n % 2 == 0` filters as it builds; the dict version uses
`{key: value for ...}`. They're not just shorter - they signal "I'm transforming a collection," easier to
read than a loop whose purpose you must infer.

⚠️ **Gotcha - don't cram everything in.** A comprehension with two filters and a nested loop is *worse*
than a plain loop. Use them for simple transform-and-filter; reach for a loop as logic grows.

### Unpacking - pull a sequence apart into names

```python runnable
point = (3, 4)
x, y = point                       # unpack a tuple into two names
first, *rest = [1, 2, 3, 4]        # * grabs "everything else"
print(x, y)
print(first, rest)
```
```console
$ python unpack.py
3 4
1 [2, 3, 4]
```
*What just happened:* `x, y = point` assigned `3` and `4` in one move; `*rest` swept the rest into a
list. This is also how Python returns "multiple values" - a function returns a tuple and the caller
unpacks it.

### enumerate and zip - loop like a Python programmer

**What they are.** `enumerate` gives you the index *and* item while looping; `zip` walks two (or more)
sequences in lockstep. Both replace clunky index bookkeeping.

```python runnable
names = ["Ana", "Bo", "Cy"]
scores = [90, 85, 95]

for i, name in enumerate(names):           # index + value, no counter needed
    print(i, name)

for name, score in zip(names, scores):     # two lists, paired up
    print(f"{name}: {score}")
```
```console
$ python loops.py
0 Ana
1 Bo
2 Cy
Ana: 90
Bo: 85
Cy: 95
```
*What just happened:* `enumerate(names)` yielded `(0, "Ana")`, `(1, "Bo")`, ... - no manual `i = 0; i +=
1` counter needed. `zip(names, scores)` yielded `("Ana", 90)`, `("Bo", 85)`, ..., pairing the lists
position by position. ⚠️ Different-length lists? `zip` stops at the shorter one, quietly.

### Truthiness - empty things are False

**What it actually is.** Python lets you test collections directly: an empty list, string, dict, `0`,
and `None` are all "falsy"; non-empty ones are "truthy." So write `if items:`, not `if len(items) > 0:`.

```python runnable
items = []
if items:
    print("has stuff")
else:
    print("empty")            # this runs - [] is falsy
```
```console
$ python truthy.py
empty
```
*What just happened:* The empty list counted as `False`, so `else` ran. `if items:` reads as "if there
are any items," exactly what you mean. ⚠️ One catch: it can't tell `None` (missing) apart from `[]`
(present but empty), or `0` from absent. When that matters, test explicitly: `if value is not None:`.

### Context managers - `with` for guaranteed cleanup

You met `with open(...)` in [Phase 7](07-errors-and-io.md). The idiom generalizes: whenever something
must be set up and reliably torn down - a file, a network connection, a lock - `with` is the Pythonic
way, since cleanup happens even if an exception fires inside.

## The gotcha cheat-card

> **Read these once and you'll dodge an afternoon of confusion each. They bite nearly everyone.**

| The gotcha | What actually happens | The fix |
|---|---|---|
| **Mutable default argument** - `def f(x, items=[]):` | The `[]` is created *once*, at function definition, and *shared across every call* - it accumulates between calls. | Default to `None`, build inside: `def f(x, items=None): items = items or []` |
| **Late-binding closures** - functions made in a loop all see the *final* loop value | The closure captures the *variable*, not its value at creation time - all read `i` after the loop ended. | Bind it as a default arg: `lambda i=i: ...`, or use a factory function |
| **`is` vs `==`** - `a is b` for comparing values | `is` checks "same object in memory," not "equal value" - works by accident for small ints/`None`, fails on bigger values. | Use `==` for value equality; reserve `is` for `is None` / `is True` |
| **Integer caching** - `a is b` "works" for `256` but not `257` | CPython pre-caches small ints (−5 to 256) as shared objects, so `is` happens to be `True` for them; above that, separate objects. | Same fix: never use `is` to compare numbers - use `==` |
| **The GIL** - threads don't speed up CPU work | CPython's Global Interpreter Lock lets only one thread run Python bytecode at a time, so CPU-bound threads never run in parallel. | Use `multiprocessing` (or a native lib) for CPU work; threads suit I/O-bound waiting |
| **Shadowing stdlib names** - naming a file `random.py` or a variable `list` | `random.py` shadows the standard library's, so `import random` imports *your* file; `list = [...]` hides the built-in `list()`. | Don't name files/variables after stdlib modules or built-ins |

Two of these deserve seeing in code:

**Mutable default argument:**
```python runnable
def add_item(item, basket=[]):     # the trap
    basket.append(item)
    return basket

print(add_item("a"))
print(add_item("b"))               # surprise: "a" is still here
```
```console
$ python default.py
['a']
['a', 'b']
```
*What just happened:* The default `[]` was created *once* at definition time and reused on every call
omitting `basket`, so the second call appended to the *same* list, which still held `"a"`. Fix: default
to `None` and create a fresh list inside the function each call.

**`is` vs `==`:**
```python runnable
a = int("257")     # built at runtime, not a shared literal
b = int("257")
print(a == b)      # equal value?  yes
print(a is b)      # same object?  no (above the cached range)
```
```console
$ python identity.py
True
False
```
*What just happened:* `a` and `b` hold equal values, so `==` is `True`. But they're separate integer
objects in memory (257 is past CPython's small-int cache), so `is` - "the *same* object?" - is `False`.
We built them with `int("257")` on purpose: two bare `257` literals in one file get folded into a single
shared object by the compiler, so `is` would sneakily report `True` there - one more reason never to
trust `is` on numbers. Use `==` for values, `is` for `None`/`True`/`False`.

> 💡 **Key point.** Almost every gotcha here comes from one of two confusions: *when* something is
> created (default args, closures - timing), or *what* equality means (`is` vs `==` - identity vs
> value). Hold those two distinctions and the surprises mostly evaporate.

## Recap

1. **Comprehensions** build lists/dicts in one readable expression; keep them simple.
2. **Unpacking** (`x, y = point`, `first, *rest = ...`) pulls sequences apart by name.
3. **`enumerate`** gives index+item; **`zip`** walks sequences in lockstep (stopping at the shortest).
4. **Truthiness** lets you write `if items:` - test `is not None` when empty and missing differ.
5. **`with`** (context managers) guarantees cleanup, exceptions or not.
6. The **gotcha cheat-card** - mutable defaults, late-binding closures, `is` vs `==`, integer caching,
   the GIL, shadowing stdlib names - are the surprises that bite everyone once. Now they won't bite you.

That's the *basics* done - phases 1-9. From here the guide goes deeper, into how Python actually works
under your code, starting with its object model.


---

# The Data Model & Dunder Methods

Back in Phase 6 you met `__init__` and `self`, and learned the word **dunder** - double underscore.
You were told these names are "hooks": you define them, and Python calls them at the right moment.

Here's the secret nobody tells you up front: **Python's built-in syntax is mostly an illusion.**
`len(x)` doesn't come from privileged knowledge of how long things are - it calls `x.__len__()`. `a + b`
calls `a.__add__(b)`. `print(x)` calls `x.__repr__()` or `x.__str__()`. The square brackets, the `+`,
the `==`, the `for` loop - all of it is sugar over method calls on *your* objects. Learn which method
each piece of syntax calls, and your own classes can behave exactly like the built-ins.

This collection of hooks has a name: **the data model**.

## The one rule: syntax dispatches to a dunder method

**What it actually is.** Almost every piece of Python syntax delegates to a corresponding dunder method.
The syntax is the friendly face; the dunder does the work. You never call the dunder yourself - you write
the syntax, and Python calls it on the object involved.

📝 **The data model** - Python's published contract of which dunder each operator, built-in, and piece of
syntax calls. Implement the right dunders and your object plugs into the language as if it were built in.

Here's the dispatch for one example, `len(x)`:

```mermaid
flowchart LR
  call["len(x)"] --> dispatch["Python looks up<br/>x.__len__"]
  dispatch --> method["x.__len__()"]
  method --> result["returns an int"]
```

*One idea:* `len()` is a thin wrapper that finds your object's `__len__` method and calls it. The same
shape holds for the rest of the model - find the syntax, find the dunder behind it.

A handful of the most common pairings, so the pattern is concrete:

| You write | Python calls |
|---|---|
| `len(x)` | `x.__len__()` |
| `x[key]` | `x.__getitem__(key)` |
| `a + b` | `a.__add__(b)` |
| `a == b` | `a.__eq__(b)` |
| `print(x)` / `str(x)` | `x.__str__()` |
| `repr(x)` / the REPL echo | `x.__repr__()` |
| `for item in x:` | `x.__iter__()` |
| `x()` | `x.__call__()` |

> 💡 **Key point.** You won't memorize this table, and you don't need to. Hold the *rule* - "syntax calls
> a dunder on the object" - and look up the specific name when you need it.

## `__repr__` vs `__str__` - how your object describes itself

**The problem it solves.** Make a class and print an instance, and you get this:

```python
class Money:
    def __init__(self, cents):
        self.cents = cents

print(Money(500))
```
```console
$ python money.py
<__main__.Money object at 0x10f4c2a90>
```
*What just happened:* Python had no idea how to turn your `Money` into text, so it fell back to its
default: the class name and the object's memory address - useless to anyone reading a log at 2am.

You teach it with two dunders, and the difference between them matters:

📝 **`__repr__`** - the *unambiguous* representation, aimed at a developer. Ideally it looks like the code
that would recreate the object: `Money(500)`. This is what the REPL echoes and what shows up in a list, a
dict, or a debugger.
📝 **`__str__`** - the *readable* representation, aimed at an end user. `print()` and `str()` use it; if
undefined, Python falls back to `__repr__`.

```python runnable
class Money:
    def __init__(self, cents):
        self.cents = cents

    def __repr__(self):
        return f"Money({self.cents})"          # looks like the constructor call

    def __str__(self):
        return f"${self.cents / 100:.2f}"      # looks like money

m = Money(500)
print(m)            # uses __str__
print(repr(m))      # uses __repr__
print([m])          # a list shows __repr__ of its contents
```
```console
$ python money.py
$5.00
Money(500)
[Money(500)]
```
*What just happened:* `print(m)` used `__str__` for the human-friendly `$5.00`; `repr(m)` used `__repr__`
for `Money(500)` - readable enough to paste back into code. Collections always show their elements'
`__repr__`, which is why a good one makes debugging calmer: `[Money(500), Money(150)]`, not three memory
addresses.

⚠️ **Gotcha - skipping `__repr__` is a debugging tax you pay forever.** Define nothing, and every log
line, error message, and `print` of a list of your objects shows `<Money object at 0x...>`. **Always give
a class a `__repr__`** - it's the single highest-value dunder. `__str__` is optional (it falls back to
`__repr__`); `__repr__` is the one you owe yourself.

## `__eq__` - what "equal" means for your object (and why it drags `__hash__` along)

**The problem it solves.** By default, two objects are equal only if they're the *same object in memory*
- the `is`-style identity from [Phase 9](09-idioms-and-gotchas.md). You almost never mean that: two
`Money(500)` values are *the same amount of money*, even as separate objects.

`__eq__` lets you define equality by *value* - Python calls it for `==`.

```python runnable
class Money:
    def __init__(self, cents):
        self.cents = cents

    def __repr__(self):
        return f"Money({self.cents})"

    def __eq__(self, other):
        return self.cents == other.cents       # equal when the amounts match

print(Money(500) == Money(500))    # same amount
print(Money(500) == Money(150))    # different amount
```
```console
$ python eq.py
True
False
```
*What just happened:* `Money(500) == Money(500)` is now `True` even though they're two distinct objects,
because your `__eq__` compares `cents` instead of identity.

But you just broke something quietly: the moment you define `__eq__`, Python *removes* the default
`__hash__`, so your objects become unhashable - they can't go in a `set` or be used as `dict` keys:

```python
amounts = {Money(500), Money(150)}
```
```console
$ python eq.py
TypeError: unhashable type: 'Money'
```
*What just happened:* Python requires that anything used as a set member or dict key have a `__hash__`,
and **objects that are equal must have the same hash**. Redefine equality but leave the old hash in
place and that promise could break - so Python disables hashing until you say what it should be.

📝 **`__hash__`** - returns an integer Python uses to bucket the object in sets and dicts. The contract: if
`a == b`, then `hash(a) == hash(b)`. Honor it by hashing the same data your `__eq__` compares - usually as
a tuple.

```python runnable
class Money:
    def __init__(self, cents):
        self.cents = cents

    def __repr__(self):
        return f"Money({self.cents})"

    def __eq__(self, other):
        return self.cents == other.cents

    def __hash__(self):
        return hash(self.cents)        # hash the SAME data __eq__ compares

amounts = {Money(500), Money(500), Money(150)}
print(amounts)                          # the two equal 500s collapse into one
```
```console
$ python eq.py
{Money(500), Money(150)}
```
*What just happened:* With `__hash__` mirroring `__eq__`, `Money` works in a `set` again - since the two
`Money(500)` objects are equal *and* hash the same, the set treats them as one. The rule: **define
`__eq__` and `__hash__` together, over the same fields, or not at all.**

⚠️ **Gotcha - `__eq__` without `__hash__` silently costs you sets and dicts.** The `TypeError` only fires
when you actually try to put the object in a set or use it as a key, possibly far from where you wrote
`__eq__`. If your "value" object should be usable as a key (most should), add `__hash__` in the same
edit. (Exception: *mutable* objects you intend to change in place are often left deliberately unhashable,
since their hash would shift underneath the set.)

## `__getitem__` and `__iter__` - making an object indexable and loopable

The data model is what makes `x[key]` and `for item in x:` work on the built-in types, and you can opt
your own classes in.

**`__getitem__` - square brackets.** Define it, and `x[key]` calls `x.__getitem__(key)`. `key` can be
anything - an integer, a string, a slice - it's your method, you decide what it means.

```python runnable
class Playlist:
    def __init__(self, songs):
        self.songs = songs

    def __getitem__(self, index):
        return self.songs[index]        # delegate to the underlying list

p = Playlist(["Intro", "Verse", "Chorus"])
print(p[0])         # calls p.__getitem__(0)
print(p[-1])        # negative indexing, for free
```
```console
$ python playlist.py
Intro
Chorus
```
*What just happened:* `p[0]` dispatched to your `__getitem__`, which handed the work to the inner list.
You didn't subclass `list` - you implemented one hook and `Playlist` started behaving like a sequence.

**`__iter__` - the `for` loop.** When you write `for song in p:`, Python calls `p.__iter__()` to get an
**iterator**, then pulls items from it one at a time. Returning `iter(self.songs)` borrows the list's own
iterator - the simplest correct move:

```python runnable
class Playlist:
    def __init__(self, songs):
        self.songs = songs

    def __iter__(self):
        return iter(self.songs)         # hand back the list's iterator

p = Playlist(["Intro", "Verse", "Chorus"])
for song in p:
    print(song)
```
```console
$ python playlist.py
Intro
Verse
Chorus
```
*What just happened:* `for song in p:` called `p.__iter__()`, got the list's iterator back, and walked it.
Your object is now loopable. This is the *shallow* end of iteration - enough to make a class work in a
`for` loop today. The real machinery (what an iterator actually is, `__next__`, and how to write your own,
including ones that generate values lazily) is the whole of
[Iterators & Generators](11-iterators-and-generators.md).

## Operator overloading - teaching `+` to your type

**What it actually is.** **Operator overloading** is the data-model rule applied to math symbols: `a + b`
calls `a.__add__(b)`, `a - b` calls `a.__sub__(b)`, and so on. The operators carry no built-in knowledge
of *your* type; you supply the meaning by defining the dunder.

Combine this with the `__repr__` and `__eq__` from before and `Money` becomes a small, complete value
type:

```python runnable
class Money:
    def __init__(self, cents):
        self.cents = cents

    def __repr__(self):
        return f"Money({self.cents})"

    def __str__(self):
        return f"${self.cents / 100:.2f}"

    def __eq__(self, other):
        return self.cents == other.cents

    def __hash__(self):
        return hash(self.cents)

    def __add__(self, other):           # called for  self + other
        return Money(self.cents + other.cents)

    def __mul__(self, factor):          # called for  self * factor
        return Money(self.cents * factor)

price = Money(500)
tax = Money(45)
total = price + tax                     # __add__
doubled = price * 2                     # __mul__
print(total)
print(doubled)
print(total == Money(545))
```
```console
$ python money.py
$5.45
$10.00
True
```
*What just happened:* `price + tax` dispatched to `__add__`, which returned a *new* `Money` of 545 cents.
`price * 2` dispatched to `__mul__`. Then `total == Money(545)` used `__eq__`. `Money` now reads exactly
like a built-in number type - `+`, `*`, `==`, a clean `print` - but every behavior is a method you wrote.
That's the payoff of the data model: **your types become first-class citizens of the language.**

📝 **Return a new object, don't mutate.** `__add__` returns a fresh `Money` rather than changing `self` -
mirroring how `+` works everywhere in Python (`3 + 4` doesn't change `3`) and keeping your type
predictable. Operators that quietly mutate their operands surprise everyone who uses them.

> 🪖 **War story.** A teammate built a `Vector` class for a physics sim and made `+` mutate the left-hand
> vector in place to "save an allocation." A week later, positions were drifting - `total = a + b` was
> silently corrupting `a` every frame. The fix was one line: return a new `Vector`. Operators are expected
> to be pure; honor the reader's intuition over the micro-optimization.

## Recap

1. **The data model is one rule:** syntax dispatches to a dunder on the object. `len(x)` → `x.__len__()`,
   `a + b` → `a.__add__(b)`, `x[k]` → `x.__getitem__(k)`. Learn the rule, look up the name.
2. **`__repr__` vs `__str__`:** `__repr__` is the unambiguous, developer-facing form (also what lists and
   the REPL show); `__str__` is the readable form `print` uses. **Always define `__repr__`** - skipping it
   gives you the useless `<object at 0x...>` in every log.
3. **`__eq__` defines value equality** (`==`), but makes your object **unhashable** until you add
   **`__hash__`** - define the two together over the same fields, or sets and dicts break.
4. **`__getitem__`** makes `x[key]` work; **`__iter__`** makes `for item in x:` work. (The deep version of
   iteration is next.)
5. **Operator overloading** is the same rule for symbols: define `__add__`, `__mul__`, etc. Return *new*
   objects; don't mutate operands.

You can now make your own classes behave like Python's built-ins. `__iter__` was touched only lightly -
iteration is deep enough, and useful enough, to deserve its own phase. Next: what an iterator really is,
and how generators produce values lazily, one at a time, without building the whole sequence in memory.

Quick check - make sure these stuck:

```quiz
[
  {"q":"When you write len(x), what does Python actually do?","choices":["It reads a hidden length field that every object stores","It calls x.__len__() - len() is a thin wrapper over that dunder","It counts the object's attributes in memory"],"answer":1,"explain":"The data model is one rule: syntax dispatches to a dunder. len(x) finds and calls x.__len__(); it has no privileged knowledge of length."},
  {"q":"Why is defining __repr__ on your class considered the single highest-value dunder?","choices":["Without it, print(x), logs, and lists of your objects all show a useless <object at 0x...>","It is required before you can use == on instances","It makes the object hashable"],"answer":0,"explain":"__repr__ is the developer-facing form shown by the REPL, debuggers, and inside collections. Skip it and every log line shows a memory address instead of something readable."},
  {"q":"You add __eq__ to a class so two equal-valued instances compare equal. What breaks?","choices":["Nothing - __eq__ is fully self-contained","The instances become unhashable: defining __eq__ removes the default __hash__, so they can't go in a set or be dict keys until you add __hash__","print() stops working on them"],"answer":1,"explain":"Equal objects must hash equally, so once you redefine equality Python disables the inherited hash. Define __eq__ and __hash__ together over the same fields, or sets and dicts break."}
]
```


---

# Iterators & Generators

You've written `for line in file:` and `for x in my_list:` since Phase 2, and it just worked. But what is
the `for` loop actually *doing*? And what happens when a list gets so big it eats all your RAM? This phase
answers both, and the idea connecting them is one of the most useful in Python.

That idea is **laziness**: producing values *one at a time, on demand*, instead of building the whole
collection up front. A list of a billion numbers needs a billion numbers' worth of memory. A *lazy*
sequence of a billion numbers needs room for one. Once that clicks, you can process a 10 GB file on a
laptop, or loop over an *infinite* sequence without your machine catching fire.

## The iterator protocol - what a `for` loop really does

**What it actually is.** An **iterable** is anything you can loop over (a list, a string, a file, a dict).
An **iterator** actually walks through it, handing you one item at a time and remembering where it left
off. Two roles: the iterable is the book; the iterator is the bookmark.

📝 **Iterable** - something you *can* loop over (has `__iter__`). **Iterator** - the stateful walker that
produces items one by one (has `__next__`). `for` asks the iterable for a fresh iterator, then pulls
items until they run out.

Writing `for x in things:` makes Python do three things under the hood:

1. Calls `iter(things)` to get an iterator (this runs `things.__iter__()`).
2. Calls `next(...)` on that iterator over and over to get each value (this runs `__next__()`).
3. Stops when `__next__` raises a special exception, `StopIteration`, which means "nothing left."

```mermaid
flowchart LR
  A[for x in things] --> B["iter(things)<br/>__iter__"]
  B --> C["next(it)<br/>__next__"]
  C -->|value| D[run loop body]
  D --> C
  C -->|StopIteration| E[loop ends]
```

*One idea:* the `for` loop is an automatic `next()`-calling machine. It keeps asking for the next value
and stops the moment the iterator signals it's empty.

**A real example.** Drive that machinery by hand to see it move:

```python runnable
things = ["a", "b"]
it = iter(things)          # get an iterator (the bookmark)

print(next(it))            # pull the first item
print(next(it))            # pull the second
try:
    print(next(it))        # nothing left...
except StopIteration:
    print("done - StopIteration raised")
```
```console
$ python protocol.py
a
b
done - StopIteration raised
```
*What just happened:* `iter(things)` made an iterator that remembers its position. Each `next(it)` advanced
it by one and returned that item. The third `next(it)` found nothing left and raised `StopIteration` - the
signal a `for` loop catches to know it's time to stop. A `for` loop is just this, with the `try/except`
handled for you.

**Why this saves you later.** Once you know `for` is "call `next` until `StopIteration`," a pile of Python
behavior stops being mysterious: why a file object can be looped but not indexed, why you can't rewind a
loop mid-stream, and how generators plug straight into every `for` loop you'll write.

## Generators - a function that pauses and resumes

Writing a class with `__iter__` and `__next__` to produce a sequence is a lot of ceremony. Python's easier
way to make an iterator is the **generator**.

**What it actually is.** A **generator** is a function that uses `yield` instead of `return`. The moment a
function contains `yield`, calling it doesn't run the body - it hands back an iterator. Each time
something pulls a value, the function runs until the next `yield`, hands that value out, then **freezes
right there**, remembering all its local variables. The next pull thaws it and continues from that spot.

📝 **`yield`** - like `return`, but instead of ending the function, it pauses it and produces one value.
The function picks up where it left off on the next request. A function with `yield` in it is a generator.

**Why this exists.** `return` ends a function and throws away everything it knew. `yield` produces a value
*without* ending, so a single function can produce a whole stream over time, keeping its place between
values - exactly the "one item at a time, remember where you were" behavior the iterator protocol wants.

**A real example.** Watch the pausing happen:

```python runnable
def count_to_three():
    print("  -> starting")
    yield 1
    print("  -> resumed after 1")
    yield 2
    print("  -> resumed after 2")
    yield 3

for n in count_to_three():
    print("got", n)
```
```console
$ python gen.py
  -> starting
got 1
  -> resumed after 1
got 2
  -> resumed after 2
got 3
```
*What just happened:* Calling `count_to_three()` ran *none* of the body - it returned a generator. The
`for` loop pulled the first value, running the function to `yield 1` before it froze. Pulling again thawed
it right after that `yield`, ran to `yield 2`, and froze again. The interleaved prints prove the function
is genuinely pausing and resuming, not running all at once.

**The gotcha - a generator is single-use.** A generator is an iterator, and an iterator gets *consumed*:
once you've walked to the end, it's empty forever.

```python runnable
def squares():
    for n in range(3):
        yield n * n

gen = squares()
print("first pass: ", list(gen))   # drains it
print("second pass:", list(gen))   # already empty
```
```console
$ python single_use.py
first pass:  [0, 1, 4]
second pass: []
```
*What just happened:* The first `list(gen)` pulled every value until `StopIteration`, exhausting the
generator. The second `list(gen)` started where the first left off - at the end - so it got an empty list.
This trips people in sneaky ways: passing a generator to a function that loops over it twice, or printing
it for debugging (which drains it) before the real loop runs.

⚠️ **Fix:** if you need to iterate twice, call the generator function again for a *fresh* one
(`squares()`), or materialize it once into a list - but only if the data fits in memory, the very thing
generators exist to avoid.

## Generator expressions - comprehensions that don't build the list

You met list comprehensions in [Phase 9](09-idioms-and-gotchas.md): `[x*x for x in nums]` builds a whole
list. Swap the square brackets for parentheses and you get a **generator expression** - same syntax, but
*lazy*: it produces values one at a time and never holds the full result in memory.

**What it actually is.** `(x*x for x in nums)` is a generator written inline, computing each value only
when asked. The difference from `[x*x for x in nums]` isn't the output values - it's *when* and *whether*
they all exist at once.

```python runnable
import sys

list_comp = [x * x for x in range(10000)]    # builds all 10,000 now
gen_expr  = (x * x for x in range(10000))    # builds nothing yet

print("list comp bytes:", sys.getsizeof(list_comp))
print("gen expr bytes: ", sys.getsizeof(gen_expr))
print("first three:", next(gen_expr), next(gen_expr), next(gen_expr))
```
```console
$ python genexpr.py
list comp bytes: 85176
gen expr bytes:  208
first three: 0 1 4
```
*What just happened:* The list comprehension allocated all 10,000 squares immediately - tens of kilobytes,
growing with the input. The generator expression allocated a tiny fixed-size object holding *the recipe*,
not the results: `208` bytes whether the range is 10,000 or 10 billion. The squares only come into being
as `next()` asks for them. (Sizes are from CPython on a 64-bit build and vary by platform, but the lesson
- fixed-and-tiny vs. grows-with-input - holds everywhere.)

💡 **Key point.** Use a **list** comprehension when you need the whole collection in hand (to index it,
loop twice, or pass it around). Use a **generator** expression when you'll consume the values once, in a
single pass - especially if the input is huge. Rule of thumb: if it feeds straight into a `for`, `sum()`,
`any()`, or `min()`, a generator expression is usually leaner.

## Why it matters - huge files and infinite sequences

Here's where laziness stops being a curiosity and starts saving your program.

**Process a file bigger than your RAM.** A file object is *already* a lazy iterator over its lines -
looping it reads one line at a time, never the whole file. Wrap that in your own generator to build a
processing pipeline that stays small no matter the file size:

```python runnable
import io

# Pretend this is a 10 GB log file; io.StringIO behaves like an open file.
fake_file = io.StringIO("error: disk full\ninfo: started\nerror: timeout\n")

def error_lines(f):
    for line in f:                 # one line at a time - never the whole file
        if line.startswith("error"):
            yield line.rstrip()

for line in error_lines(fake_file):
    print(line)
```
```console
$ python bigfile.py
error: disk full
error: timeout
```
*What just happened:* `error_lines` is a generator. It pulls one line from the file, and if it's an error,
yields it - then pauses until the next line is needed. At no point does the whole file (or the whole list
of error lines) sit in memory. Point this at a real 10 GB log opened with `open(path)` and it uses the same
tiny footprint, one line at a time, start to finish. That's the headline use case for generators.

**Loop over something infinite.** A list can't be infinite - you can't store endless items. A generator
can *describe* an endless sequence and produce it on demand. You just need a way to stop pulling.

```python runnable
def naturals():
    n = 0
    while True:        # never ends on its own
        yield n
        n += 1

gen = naturals()
first_five = [next(gen) for _ in range(5)]   # pull exactly 5, then stop asking
print(first_five)
```
```console
$ python infinite.py
[0, 1, 2, 3, 4]
```
*What just happened:* `naturals()` would yield numbers forever if you let it - the `while True` never
finishes. But nothing is computed until you ask, so you pulled exactly five values with `next()` and
walked away. The generator is paused mid-`while`, holding `n`, ready to continue if you come back.
⚠️ Never write a bare `for x in naturals():` with no stopping condition - it runs until you kill the
process. You need something that caps how many values you pull, which is what `itertools` gives you.

## A peek at `itertools` - the lazy toolkit

The standard library's `itertools` module is a box of ready-made lazy building blocks. They take iterators
and produce iterators without materializing anything. Three you'll reach for constantly:

- **`count(start)`** - counts upward forever (a lazy, infinite `range`).
- **`islice(it, n)`** - takes the first `n` items from any iterator, then stops. Your "stop pulling" tool
  for infinite generators.
- **`chain(a, b)`** - glues iterables end to end into one stream, without copying them into a combined list.

```python runnable
from itertools import count, islice, chain

# islice tames an infinite counter: take 5, no manual next() loop
first_five = list(islice(count(0), 5))
print(first_five)

# chain walks two sequences as one, lazily
for x in chain([1, 2], ["a", "b"]):
    print(x)
```
```console
$ python itertools_demo.py
[0, 1, 2, 3, 4]
1
2
a
b
```
*What just happened:* `count(0)` is an endless lazy counter; `islice(count(0), 5)` pulled exactly the
first five values and raised `StopIteration`, so `list(...)` got `[0, 1, 2, 3, 4]` and the counter never
ran away. `chain([1, 2], ["a", "b"])` produced all four items as a single stream without building a
combined `[1, 2, "a", "b"]` list. Same pattern throughout: take iterators, return iterators, stay lazy.

## Recap

1. A `for` loop calls `iter()` to get an **iterator**, then `next()` until it raises **`StopIteration`**.
   That's the whole **iterator protocol**.
2. A **generator** is a function with **`yield`**: it pauses and resumes, producing a stream of values one
   at a time while remembering its place.
3. A **generator expression** `(x*x for x in ...)` is a lazy comprehension - same syntax as `[...]`, but it
   never builds the full list, so its memory stays tiny no matter the input size.
4. Laziness is the payoff: **process a 10 GB file or an infinite sequence** without loading it into RAM,
   one item at a time.
5. ⚠️ A generator is **single-use** - once exhausted, it's empty. Make a fresh one (or a list) if you need
   to iterate again.
6. **`itertools`** (`count`, `islice`, `chain`) gives you lazy building blocks; `islice` is how you safely
   take a finite slice of an infinite generator.

You can now produce values without paying for all of them up front. Next: **decorators** - wrapping a
function in extra behavior without touching its body, built on the same "functions are just objects" idea
that makes generators tick.

## Quick check

Test yourself on the one idea that makes this whole phase tick - laziness:

```quiz
[
  {
    "q": "What does `yield` do that `return` doesn't?",
    "choices": [
      "Pauses the function and produces a value, then resumes from that exact spot on the next request",
      "Ends the function and discards its local variables, just like return",
      "Builds and returns a complete list of all values at once",
      "Makes the function run faster by skipping the iterator protocol"
    ],
    "answer": 0,
    "explain": "`yield` produces a value without ending the function. It freezes the function in place, remembering all local variables, and thaws it on the next pull - that's what lets one function emit a whole stream over time."
  },
  {
    "q": "Why does `(x*x for x in range(10_000_000))` use far less memory than `[x*x for x in range(10_000_000)]`?",
    "choices": [
      "The generator expression computes values one at a time on demand instead of building the full list up front",
      "The generator expression secretly uses a faster C loop",
      "Parentheses are always cheaper than square brackets in Python",
      "The generator expression rounds the numbers to save space"
    ],
    "answer": 0,
    "explain": "A generator expression holds the recipe, not the results. It produces each value only when asked, so its footprint stays tiny and fixed no matter how big the input is - while the list comprehension allocates every value immediately."
  },
  {
    "q": "You write `gen = squares()`, then `list(gen)` twice in a row. What does the second `list(gen)` return?",
    "choices": [
      "An empty list `[]` - the generator was exhausted by the first pass",
      "The same list as the first call - generators restart automatically",
      "An error, because you can't call `list()` on a generator twice",
      "Half the values, because the generator remembers only its midpoint"
    ],
    "answer": 0,
    "explain": "A generator is single-use. The first `list(gen)` drains it to `StopIteration`, leaving it empty forever. To iterate again, call the generator function for a fresh one (`squares()`), or materialize the values into a list first."
  }
]
```


---

# Decorators

The first time you see `@app.get("/")` or `@property` sitting on top of a function, it looks like a
spell - the function gets *powers* it didn't ask for, hidden behind a symbol you've never been formally
introduced to. People copy these incantations from tutorials for years without knowing what `@` does.

Here's the relief up front: there is no magic. A decorator is an ordinary function. The `@` is a
two-character shortcut for one line of plain Python you could write yourself. By the end of this phase
you'll be able to read any `@something` and know exactly what it's doing - and write your own.

To get there we need one idea first - the one most languages never make you think about.

## Functions are first-class objects

**What it actually is.** In Python, a function is *a value*, just like `3` or `"hello"` or a list. The
`def` statement doesn't summon something special - it creates an object and binds a name to it. Because a
function is a value, you can do everything to it you'd do to any value: store it in a variable, put it in
a list, pass it to another function, return it from one.

**Why this matters here.** Decorators are *built* out of this single fact: if a function can be passed
around and returned like data, a function can take *another function* as input and hand back a new one.
That's the whole game. Prove each piece in code.

```python runnable
def shout(text):
    return text.upper() + "!"

# 1. A function is a value - bind it to another name
yell = shout
print(yell("hello"))

# 2. Pass a function as an argument
def apply_twice(func, value):
    return func(func(value))

print(apply_twice(shout, "hi"))
```
```console
$ python firstclass.py
HELLO!
HI!!
```
*What just happened:* `yell = shout` didn't *call* `shout` - no parentheses. It bound a second name to the
same function object, so `yell` and `shout` are now two names for one thing. `apply_twice` received
`shout` as plain data in its `func` parameter and called it - a function went *into* another function as
an argument and got used there. (`apply_twice(shout, "hi")` runs `shout(shout("hi"))` → `shout("HI!")` →
`"HI!!"`.)

📝 **First-class** - a value the language lets you pass, return, and store with no special ceremony. In
Python, functions are first-class (so are classes and modules) - this is *why* decorators are possible.

### Functions can be defined inside, and returned out

The last piece: a function can be *defined inside another function*, and the outer function can *return*
it as its result.

```python runnable
def make_greeter(greeting):
    def greet(name):                     # defined inside make_greeter
        return f"{greeting}, {name}!"
    return greet                         # hand the inner function back

say_hello = make_greeter("Hello")        # this returns a function
say_hi = make_greeter("Hi")
print(say_hello("Ana"))
print(say_hi("Bo"))
```
```console
$ python factory.py
Hello, Ana!
Hi, Bo!
```
*What just happened:* `make_greeter("Hello")` ran, defined a fresh `greet` inside, and *returned* it
without calling it - so `say_hello` is now a function. The clever part: the returned `greet` still
remembers the `greeting` it was born with (`"Hello"`), even though `make_greeter` already finished. That
remembered value is a **closure** - the inner function closes over the variables it used from the outer
one.

📝 **Closure** - an inner function plus the outer-scope variables it captured. `say_hello` carries
`greeting="Hello"` around with it forever; `say_hi` carries `"Hi"`. Same code, different captured data.

Combine those two abilities - *pass a function in*, *return a function out* - and you have a decorator.

## A decorator is a function that takes a function and returns a function

**What it actually is.** A **decorator** is a function whose input is a function and whose output is a
(usually wrapped) function. You give it `f`; it gives back a new function that does something *extra*
around `f` - runs code before, after, or instead of it - then hands control to `f` and returns its result.

> 💡 **Key point.** That one sentence is the entire concept: *takes a function, returns a function.*
> Everything else - the `@`, the arguments, `functools.wraps` - is detail layered on top of it.

**A real example.** Here's a decorator that logs every call. Read it bottom-up: `log_calls` takes a
function `func`, defines a `wrapper` that prints around calling `func`, and returns `wrapper`.

```python runnable
def log_calls(func):
    def wrapper(*args, **kwargs):
        print(f"-> calling {func.__name__}")
        result = func(*args, **kwargs)
        print(f"<- {func.__name__} returned {result}")
        return result
    return wrapper

def add(a, b):
    return a + b

add = log_calls(add)          # wrap it by hand - no @ yet
print(add(2, 3))
```
```console
$ python logcalls.py
-> calling add
<- add returned 5
5
```
*What just happened:* `log_calls(add)` returned `wrapper` - a brand-new function that closes over the
original `add` (as `func`). We rebound the name `add` to point at that wrapper, so calling `add(2, 3)`
really calls `wrapper(2, 3)`: it prints, calls the *real* `add` through `func`, prints again, and returns
the result. The original function never changed - it got *wrapped*.

📝 **`*args, **kwargs`** - "accept any positional arguments and any keyword arguments." The wrapper uses
this so it works for *any* function regardless of its signature, then forwards everything to `func`
unchanged. You met functions in [Phase 4](04-control-flow-and-functions.md); this is where treating them
as values pays off.

## `@decorator` is sugar for `f = decorator(f)`

Now the reveal: that line `add = log_calls(add)` is exactly what `@` does, just written above the
function instead of below it.

These two snippets are *identical* in behavior:

```python
# The longhand you already understand:
def add(a, b):
    return a + b
add = log_calls(add)

# The @ shorthand - same thing, said once:
@log_calls
def add(a, b):
    return a + b
```

The `@log_calls` line means: "after defining `add`, immediately run `add = log_calls(add)`." That's the
whole definition of `@` syntax - not a keyword with hidden rules, just a rewrite.

```mermaid
flowchart LR
  A["@d above def f"] --> B["f = d(f)"]
  B --> C["name f now points at<br/>d's returned wrapper"]
```

*One idea:* `@d` is a textual shortcut. The interpreter defines your function, passes it through `d`, and
rebinds the name to whatever `d` returns. Knowing this, you can read any decorator stack.

Here it is doing real work - `@timer` reports how long a function took:

```python runnable
import time

def timer(func):
    def wrapper(*args, **kwargs):
        start = time.perf_counter()
        result = func(*args, **kwargs)
        elapsed = time.perf_counter() - start
        print(f"{func.__name__} took {elapsed:.4f}s")
        return result
    return wrapper

@timer
def slow_sum(n):
    return sum(range(n))

print(slow_sum(1_000_000))
```
```console
$ python timer.py
slow_sum took 0.0143s
499999500000
```
*What just happened:* `@timer` rewrote `slow_sum` into `timer(slow_sum)` - the wrapper that times the
call. We never touched `slow_sum`'s body; the timing logic lives entirely in the decorator and can be
slapped onto any function the same way. (Your exact seconds will differ - it's a live measurement.)

## `functools.wraps` - don't lose the function's identity

There's a quiet bug in every decorator written so far. When you wrap a function, the *name* now points at
`wrapper` - so the original function's `__name__` and docstring vanish, replaced by the wrapper's. Tools
that introspect functions (debuggers, docs generators, test frameworks) suddenly see `wrapper` everywhere.

```python runnable
def log_calls(func):
    def wrapper(*args, **kwargs):
        return func(*args, **kwargs)
    return wrapper

@log_calls
def add(a, b):
    "Add two numbers."
    return a + b

print(add.__name__)        # we'd hope for "add"
print(add.__doc__)         # we'd hope for "Add two numbers."
```
```console
$ python noidentity.py
wrapper
None
```
*What just happened:* `add` is now the `wrapper` function, so `add.__name__` correctly reports `"wrapper"`
and the docstring is gone - `wrapper` never had one. Every function decorated with `log_calls` would
*claim to be named* `wrapper`, which is confusing in tracebacks and breaks tools that rely on `__name__`.

⚠️ **Gotcha - always wrap your wrapper with `functools.wraps`.** `functools.wraps(func)` is a decorator
you put on the *inner* wrapper. It copies the original function's `__name__`, `__doc__`, and other
metadata onto the wrapper. Leaving it out is the single most common decorator mistake.

```python runnable
import functools

def log_calls(func):
    @functools.wraps(func)             # copy func's identity onto wrapper
    def wrapper(*args, **kwargs):
        return func(*args, **kwargs)
    return wrapper

@log_calls
def add(a, b):
    "Add two numbers."
    return a + b

print(add.__name__)
print(add.__doc__)
```
```console
$ python wraps.py
add
Add two numbers.
```
*What just happened:* `@functools.wraps(func)` ran on `wrapper` and copied `add`'s name and docstring onto
it. The decorated `add` *still* introspects as `add` - same name, same docs - even though it's secretly
the wrapper. Make this a reflex: every decorator gets a `@functools.wraps(func)` on its inner function.

## Decorators that take arguments

So far `@timer` and `@log_calls` take no configuration. But you've surely seen `@app.get("/users")` or
`@retry(times=3)` - decorators *with parentheses and arguments*. How does that work, when a decorator is
supposed to take a function?

**The trick: one more layer.** `@repeat(3)` is two steps. First Python evaluates `repeat(3)` - a normal
function call that must *return a decorator*. Then that returned decorator gets applied to your function.
So a decorator-with-arguments is a function that returns a decorator that returns a wrapper - three
layers deep.

📝 **Decorator factory** - a function you call (`repeat(3)`) that builds and returns a decorator. The
arguments you pass configure the decorator it produces.

```python runnable
import functools

def repeat(times):                        # the factory: takes the config
    def decorator(func):                  # the actual decorator: takes the function
        @functools.wraps(func)
        def wrapper(*args, **kwargs):     # the wrapper: takes the call's args
            for _ in range(times):
                result = func(*args, **kwargs)
            return result
        return wrapper
    return decorator

@repeat(3)
def greet(name):
    print(f"Hi, {name}!")

greet("Ana")
```
```console
$ python repeat.py
Hi, Ana!
Hi, Ana!
Hi, Ana!
```
*What just happened:* `@repeat(3)` ran in two beats. `repeat(3)` was called first and returned `decorator`
(carrying `times=3` in a closure). Then `decorator` was applied to `greet`, exactly like a normal
decorator, producing `wrapper`. Calling `greet("Ana")` loops the wrapper three times. The extra layer
exists purely to capture the `3` before the function ever shows up.

Read `@repeat(3)` as `greet = repeat(3)(greet)`: the first call eats the argument, the second eats the
function. Once you see those two calls, decorators with arguments stop being mysterious.

## Where you've already met decorators

You don't usually *write* decorators day to day - you *use* ones other people wrote. Now that you know
the mechanism, these all read as plain code:

- **Web framework routes** - `@app.get("/users")` in FastAPI, `@app.route("/")` in Flask. The decorator
  takes your view function and registers it in the framework's routing table, then returns it (often
  unchanged). The "magic" is just `your_view = app.get("/users")(your_view)`.
- **`@property`** - a built-in decorator that turns a method into a *computed attribute*, so you can write
  `obj.area` (no parentheses) and have it run a method behind the scenes.
- **`@functools.lru_cache`** - wraps a function so its results are *cached*: call it again with the same
  arguments and get the stored answer instead of recomputing. It's a decorator factory
  (`@lru_cache(maxsize=128)`) doing exactly the layered trick you just learned.
- **`@staticmethod` / `@classmethod`** - built-in decorators that change how a method receives its first
  argument, from [Phase 6](06-objects-and-classes.md)'s class machinery.

Here's `lru_cache` turning a painfully slow recursive Fibonacci into an instant one, with a single line:

```python runnable
import functools

@functools.lru_cache(maxsize=None)
def fib(n):
    if n < 2:
        return n
    return fib(n - 1) + fib(n - 2)

print(fib(40))
print(fib.cache_info())
```
```console
$ python cache.py
102334155
CacheInfo(hits=38, misses=41, maxsize=None, currsize=41)
```
*What just happened:* `@lru_cache` wrapped `fib` so each `(n,)` it's ever called with is computed once
and remembered. Without it, `fib(40)` would recompute the same sub-results billions of times; with it,
each of the 41 distinct calls runs once (`misses=41`) and the rest are served from cache (`hits=38`). One
decorator line bought all of that, and `cache_info()` is a bonus method the decorator attached to your
function.

**Why this saves you later.** Decorators are how Python lets a library add behavior to *your* code without
you editing it - logging, timing, caching, routing, access control, retries. Read `@` as "pass this
function through that function and rebind the name" and every framework you pick up gets less mysterious.

## Recap

1. **Functions are first-class** - you can store, pass, and return them. Decorators are built entirely on
   this.
2. A **closure** is an inner function plus the outer variables it captured; returning an inner function
   keeps those values alive.
3. A **decorator** is a function that takes a function and returns a (usually wrapped) function.
4. **`@decorator`** above a `def` is pure sugar for `f = decorator(f)` - a rewrite, not magic.
5. **`functools.wraps(func)`** on the inner wrapper copies the original's name and docstring across - omit
   it and your function forgets who it is. Make it a reflex.
6. **Decorators with arguments** add one layer: `@repeat(3)` means `repeat(3)(func)` - a factory returns
   the decorator, which returns the wrapper.
7. You've already used decorators everywhere - `@app.get(...)`, `@property`, `@lru_cache`. Now you know
   exactly what each one does.

You can now read and write the `@` with confidence. Next, another piece of "magic" that turns out to be a
plain protocol: the `with` statement and the context managers behind it, the reliable way to guarantee
setup and cleanup.

See an LRU cache fill and evict, live:

```playground-lru
```

Quick check - make sure these stuck:

```quiz
[
  {
    "q": "What is a decorator, fundamentally?",
    "choices": [
      "A keyword that adds hidden behavior to a function",
      "A function that takes a function and returns a (usually wrapped) function",
      "A special class that wraps methods",
      "A comment the interpreter reads above a def"
    ],
    "answer": 1,
    "explain": "A decorator is just an ordinary function whose input is a function and whose output is a function. No magic - the @ is sugar on top of that one idea."
  },
  {
    "q": "The line `@log_calls` written above `def add(...):` is equivalent to which plain statement?",
    "choices": [
      "log_calls(add())",
      "add = log_calls",
      "add = log_calls(add)",
      "log_calls = add(log_calls)"
    ],
    "answer": 2,
    "explain": "`@d` above a def means: define the function, then run `f = d(f)`. So `@log_calls` over `add` is exactly `add = log_calls(add)` - a textual rewrite, not a keyword."
  },
  {
    "q": "Why put `@functools.wraps(func)` on your inner wrapper?",
    "choices": [
      "It makes the wrapped function run faster",
      "It copies the original's __name__, __doc__, and other metadata onto the wrapper so it still looks like itself",
      "It is required syntax or the decorator won't apply",
      "It caches the function's results between calls"
    ],
    "answer": 1,
    "explain": "Without it, the decorated name points at `wrapper`, so `__name__` reports \"wrapper\" and the docstring is lost. `functools.wraps` copies the original's identity (name, docstring, etc.) onto the wrapper."
  }
]
```


---

# Context Managers

Some things you open, you must close. A file. A network connection. A lock another
thread is waiting on. A database transaction someone needs committed or rolled
back. The pattern is always the same: setup, do your work, then - *no matter what
happens in the middle* - clean up.

That last part is where people get burned. The work in the middle can raise an
exception, and if it does, the cleanup line after it never runs. The file stays
open, the lock stays held, the connection leaks. You won't notice on a good day;
you'll notice at 2am when the server has run out of file handles and nobody can
log in.

Python has one tool that makes "always clean up" automatic: the `with` statement.
This phase covers what `with` actually guarantees, and how to build your own.

## The problem `with` solves

**The naive version.** You open something, use it, close it:

```python
f = open("data.txt")
data = f.read()
f.close()
```

This is fine - until `f.read()` raises (the disk hiccups, the file is malformed,
your parsing blows up). Then `f.close()` never runs, because the exception jumps
straight out of the function. The file handle leaks.

**The careful version.** The real fix is `try`/`finally` - the `finally` block
runs whether or not an exception happened:

```python
f = open("data.txt")
try:
    data = f.read()
finally:
    f.close()
```

This is *correct*. It's also four lines of ceremony around one line of real work,
and you have to write it every time you touch a resource. Forget the `try`/`finally`
once and you're back to leaking handles - and noisy code is code people skip.

**What it actually is.** A **context manager** is an object that knows how to set
itself up and, crucially, how to tear itself down - and the `with` statement wires
that teardown to run automatically, exception or not. It's the `try`/`finally`
above, packaged so you write it once and reuse it forever.

> 💡 **Key point.** `with` is a guarantee: *the cleanup runs no matter how the
> block ends* - normal finish, `return`, or exception. That guarantee is the whole
> reason context managers exist.

## `with open(...)` - the one you already know

You met this back in [Phase 7](07-errors-and-io.md). Now you can see what it's
really doing:

```python
with open("data.txt") as f:
    data = f.read()
# f is closed here - automatically
```

*What just happened:* `with open(...) as f` opened the file and handed it to `f`.
When the block ends - for *any* reason, including an exception thrown by
`f.read()` - Python calls the file's cleanup, which closes the handle. That's the
`finally` you no longer have to write. `as f` is just the name you give whatever
the context manager hands back.

Every `with open(...)` you've written was a `try`/`finally` in disguise. The same
machinery works for anything else that needs guaranteed cleanup - you just have to
teach an object how to enter and exit.

## The protocol: `__enter__` and `__exit__`

**What it actually is.** Any object with two dunder methods can be used in a
`with` statement:

- `__enter__(self)` runs at the top of the block. Its return value is what `as`
  binds to.
- `__exit__(self, ...)` runs at the bottom of the block - *always* - and that's
  where cleanup lives.

📝 **The context manager protocol** - an object is a context manager if it has
`__enter__` and `__exit__`. (Recall from [Phase 6](06-objects-and-classes.md):
dunder methods are hooks Python calls for you. You don't call `__enter__`
yourself - `with` calls it.)

Here's a tiny one that prints on enter and exit, so you can watch the protocol fire:

```python runnable
class Tag:
    def __init__(self, name):
        self.name = name

    def __enter__(self):
        print(f"<{self.name}>")        # setup
        return self                    # what `as` receives

    def __exit__(self, exc_type, exc_value, traceback):
        print(f"</{self.name}>")       # teardown - always runs

with Tag("body"):
    print("  hello")
```
```console
$ python tags.py
<body>
  hello
</body>
```
*What just happened:* Entering the `with` block called `__enter__`, which printed
the opening tag. Your code printed `hello`. Then, at the end of the block, Python
called `__exit__`, which printed the closing tag. You wrote the opening and
closing exactly once, and Python guaranteed they bracket whatever happens between.

Those three parameters on `__exit__` - `exc_type`, `exc_value`, `traceback` - are
how Python tells your cleanup *whether the block ended normally or blew up*. More
on them in the gotcha section below.

Here's the shape of it as a picture:

```mermaid
flowchart LR
  enter["__enter__<br/>(setup)"] --> body["your block"]
  body --> exit["__exit__<br/>(cleanup)"]
  body -. exception .-> exit
```

*One idea:* the body runs after setup, and the exit always runs after the body -
whether the body finished cleanly or threw (the dotted path).

## The easier way: `@contextmanager`

A whole class with two dunder methods is a lot of structure for something that's
really just "do this, then yield control, then do that." Python's `contextlib`
module gives you a shortcut: write a generator, mark it with `@contextmanager`,
and let a single `yield` split setup from teardown.

📝 **Generator** - a function that uses `yield` to pause and hand a value back to
its caller, then resume where it left off. For a context manager, everything
*before* the `yield` is setup, the yielded value is what `as` gets, and everything
*after* the `yield` is teardown.

```python runnable
from contextlib import contextmanager

@contextmanager
def tag(name):
    print(f"<{name}>")        # everything before yield = setup
    yield name                # hand control to the block; `name` goes to `as`
    print(f"</{name}>")       # everything after yield = teardown

with tag("body") as t:
    print(f"  hello from {t}")
```
```console
$ python tag_cm.py
<body>
  hello from body
</body>
```
*What just happened:* The generator ran up to `yield`, printing the open tag and
pausing. The yielded value `name` became `t` via `as`. Your block ran. When it
ended, the generator *resumed* from the `yield` and printed the close tag. One
function, one `yield`, the same guarantee - much less ceremony than the class.

For the cleanup to truly always run, the teardown belongs in a `finally`, so an
exception can't skip it. Here's the robust form - a small `Timer` that reports how
long a block took, even if it crashes partway:

```python runnable
import time
from contextlib import contextmanager

@contextmanager
def timer(label):
    start = time.perf_counter()
    print(f"{label}: start")
    try:
        yield                              # the block runs here
    finally:
        elapsed = time.perf_counter() - start
        print(f"{label}: done")            # always prints the result line

with timer("work"):
    total = sum(range(1_000_000))
print(total)
```
```console
$ python timer.py
work: start
work: done
499999500000
```
*What just happened:* Setup recorded the start time, `yield` ran your block, and
`finally` ran the teardown - so even if `sum(...)` had raised, you'd still see the
`done` line. Wrapping the post-`yield` work in `try`/`finally` is the habit that
makes a `@contextmanager` bulletproof.

> 💡 **Key point.** Two ways to build a context manager, same protocol underneath:
> a **class** with `__enter__`/`__exit__` when you need to hold state or reuse the
> object, or a **`@contextmanager` generator** when it's a simple setup-yield-
> teardown. Reach for the generator first; it's less to get wrong.

## Where this shows up in real code

The `with` statement is everywhere once you start looking, because *so many* things
need guaranteed cleanup:

- **Files** - `with open(...) as f:` closes the handle. The original example, and
  the most common.
- **Locks** - a `threading.Lock` is a context manager. `with lock:` acquires it on
  entry and releases it on exit, so it can't stay stuck even if the protected code
  raises:

  ```python
  import threading
  lock = threading.Lock()

  with lock:           # acquire
      shared_counter += 1
  # released here, even if the line above had thrown
  ```

  Without `with`, you'd `lock.acquire()` and `lock.release()` by hand - a
  forgotten or skipped `release()` is a classic deadlock, where another thread
  waits forever for a lock that's never coming back.

- **Database transactions** - most DB connection libraries make the connection (or
  a cursor) a context manager that **commits** the transaction if the block
  succeeds and **rolls it back** if an exception escapes. So `with conn:` means
  "all of this, or none of it" - exactly the all-or-nothing behavior a transaction
  should give you. (The exact object you wrap varies by library, but the pattern
  is the point.)

The thread tying them together: setup that must be matched by teardown you can't
afford to skip. That's the precise shape `with` was built for.

## ⚠️ The gotcha: `__exit__` runs on exception - and can swallow it

The whole point of `with` is that cleanup runs **even when the block raises** -
that's a feature, it's *why* the file still closes when `f.read()` blows up. But
it has a sharp edge worth understanding.

When an exception propagates out of a `with` block, Python passes its details into
`__exit__` through those three parameters, and **`__exit__`'s return value decides
what happens to the exception**:

- Return a **falsy** value (or nothing - the default `None`) → the exception
  continues propagating after cleanup. *This is what you almost always want.*
- Return **`True`** → Python treats the exception as *handled* and **swallows it**.
  Execution continues after the `with` block as if nothing went wrong.

Watch the difference. Here `__exit__` returns `True`, which makes the error vanish:

```python runnable
class Swallow:
    def __enter__(self):
        return self

    def __exit__(self, exc_type, exc_value, traceback):
        if exc_type is not None:
            print(f"caught {exc_type.__name__}: {exc_value}")
        return True            # True = "I handled it" = suppress the exception

with Swallow():
    print("about to fail")
    raise ValueError("boom")

print("we got here - the exception was swallowed")
```
```console
$ python swallow.py
about to fail
caught ValueError: boom
we got here - the exception was swallowed
```
*What just happened:* The `raise` sent a `ValueError` out of the block. Python
called `__exit__` with `exc_type=ValueError` and the exception details, ran the
cleanup (the print), and saw `return True` - so it *suppressed* the error. The
line after the `with` ran normally. Returning `None` (or nothing) would have let
that `ValueError` keep propagating and crash the program - the right behavior the
vast majority of the time.

The rule of thumb: **let exceptions through.** Your `__exit__` should clean up and
return nothing, so problems still surface. Returning `True` to swallow an exception
is a rare, deliberate choice - `contextlib.suppress` exists for cases where you
*mean* it (e.g. "delete this file if it exists, shrug if it doesn't"). Silently
eating errors you didn't mean to is how bugs hide for weeks.

> 🪖 **War story.** A teammate wrote a context manager for a flaky API client and,
> copying a snippet, left a `return True` in `__exit__`. For months, every failed
> API call inside that `with` block vanished without a trace - no error, no log,
> just silently wrong data downstream. The fix was deleting two characters. When in
> doubt, return nothing from `__exit__`.

## Recap

1. Anything you **open you must close** - files, locks, connections, transactions.
   `with` guarantees the close happens, even on exception, so you don't write
   `try`/`finally` by hand every time.
2. **`with open(...) as f:`** is a `try`/`finally` in disguise: the file closes
   when the block ends, no matter how it ends.
3. The **protocol** is two dunder methods: `__enter__` (setup; its return value is
   what `as` binds) and `__exit__` (cleanup; always runs).
4. **`@contextmanager`** turns a generator into a context manager - everything
   before `yield` is setup, everything after is teardown. Put the teardown in a
   `finally` so it can't be skipped.
5. ⚠️ `__exit__` runs **even on exception** (that's the point), and its **return
   value controls suppression**: return falsy/`None` to let the exception through
   (almost always what you want); return `True` only when you deliberately mean to
   swallow it.

You can now reach for `with` anytime something needs reliable cleanup, and build
your own when the standard ones don't fit. Next, making your code say what it means
about *types* - annotations and letting `mypy` catch whole classes of bugs before
you ever run the program.

## Quick check

See if the protocol stuck. Pick the best answer for each, then check yourself.

```quiz
[
  {
    "q": "In a `with` block, what guarantees that cleanup runs even if the code inside the block raises an exception?",
    "choices": [
      "The exception is silently caught and discarded by `with`",
      "`__exit__` is always called when the block ends - normal finish, return, or exception",
      "Python re-runs the block until it finishes without error",
      "Cleanup only runs if you wrap the block in `try`/`finally` yourself"
    ],
    "answer": 1,
    "explain": "The `with` statement calls `__exit__` no matter how the block ends. That's the whole guarantee - it's the `try`/`finally` packaged into the protocol, so the file closes or the lock releases even when an exception fires."
  },
  {
    "q": "Using `@contextmanager`, where does the setup code go and where does the teardown go relative to `yield`?",
    "choices": [
      "Everything goes before `yield`; teardown is a separate function",
      "Setup goes after `yield`, teardown before it",
      "Setup goes before `yield`, teardown after it (ideally in a `finally`)",
      "`yield` must appear twice - once for setup, once for teardown"
    ],
    "answer": 2,
    "explain": "A single `yield` splits the generator: everything before it is setup, the yielded value is what `as` receives, and everything after is teardown. Put the teardown in a `finally` so an exception in the block can't skip it."
  },
  {
    "q": "Your `__exit__` returns `True` after an exception propagates into it. What happens?",
    "choices": [
      "The exception is suppressed and execution continues after the `with` block",
      "The exception keeps propagating, but cleanup is skipped",
      "Python raises a new error because `True` is invalid",
      "Nothing changes - the return value of `__exit__` is ignored"
    ],
    "answer": 0,
    "explain": "Returning `True` tells Python the exception was handled, so it swallows it and runs on. Returning falsy/`None` (the usual choice) lets the exception keep propagating after cleanup - silently eating errors is how bugs hide for weeks."
  }
]
```


---

# Type Hints & mypy

Python lets you assign a string to a variable, then an integer, then a list, all on the same name, and it
won't blink. That flexibility is wonderful when you're sketching, and quietly terrifying when your
codebase is forty files deep and a function three layers down was *supposed* to receive a `User` but got
a `str` instead. The bug doesn't announce itself where you made the mistake - it surfaces later, in some
unrelated place, as a confusing crash.

**Type hints** buy back some of that safety without giving up Python's looseness. You annotate what a
function expects and returns, and a separate tool reads those annotations and tells you - *before you run
anything* - where the types don't line up. This is **gradual typing**: add hints where they pay off, leave
them off where they don't. No all-or-nothing rewrite.

## The one idea: hints are for humans and tools, not the interpreter

**What they actually are.** A type hint is a note attached to a variable, parameter, or return value
saying "this is meant to be a `str`" (or `int`, or `list[str]`, …) - *documentation that tools can read*.
The Python interpreter itself **does not check it and does not act on it**. Your annotated code runs
exactly the same as the unannotated version would.

📝 **Gradual typing** - the approach where some of your code is annotated and some isn't, and that's fine.
You opt in incrementally, file by file, function by function.

> 💡 **Key point.** Hold onto this one line for the whole phase: *hints change nothing at runtime.* They
> exist for three audiences - the next human who reads your code, your editor (autocomplete and inline
> warnings), and a **type checker** like mypy that reads them and flags mismatches. The interpreter
> ignores them.

Here's the flow, because it's the part people get backwards:

```mermaid
flowchart LR
  you(You write hints) --> mypy(mypy / your IDE)
  mypy -->|flags mismatches| you
  you --> py(Python interpreter)
  py -.ignores hints.-> run(runs the code)
```

*One idea:* the checker and the interpreter are two separate readers of the same file. mypy is where hints
have teeth; Python at runtime just runs.

## Basic annotations

**What they look like.** You annotate a parameter with `name: type`, and a function's return value with
`-> type` after the parentheses. You can annotate plain variables too, with `name: type = value`.

```python runnable
def greet(name: str) -> str:        # takes a str, returns a str
    return f"Hello, {name}!"

count: int = 3                       # a variable annotation
message: str = greet("Ada")

print(message)
print(count)
```
*What just happened:* `greet` is annotated to take a `str` and return a `str`, and `count` is annotated as
an `int`. The code printed exactly what it would have without any annotations - `Hello, Ada!` and `3`. The
hints added meaning for tools and readers, and **zero** change to behavior. `-> str` (the return type) is
the single most common piece of typing syntax you'll write.

📝 **Annotation** - the `: type` or `-> type` note itself. Python stores annotations on the function (in
`__annotations__`) but never consults them while running.

## Container generics - saying *what's inside*

A bare `list` tells a reader it's a list, but not a list *of what*. **Generics** say the element type:
`list[int]` is "a list of integers," `dict[str, int]` is "a dict whose keys are strings and values are
ints."

```python runnable
def total(scores: list[int]) -> int:    # a list of ints in, one int out
    return sum(scores)

ages: dict[str, int] = {"Ana": 30, "Bo": 25}    # str keys, int values

print(total([90, 85, 95]))
print(ages["Ana"])
```
*What just happened:* `list[int]` and `dict[str, int]` spelled out what the containers hold. `total` now
documents that it wants a list of integers specifically, and mypy will complain if you hand it
`["90", "85"]` (strings). The square-bracket syntax - `list[...]`, `dict[..., ...]`, `tuple[...]`,
`set[...]` - parameterizes any container.

⚠️ **Gotcha - lowercase built-ins need Python 3.9+.** Writing `list[int]` and `dict[str, int]` directly
works on 3.9 and newer. Older versions require `List[int]` / `Dict[str, int]` imported from the `typing`
module. On a modern Python, prefer the lowercase built-in forms shown here.

## When a value might be missing - `Optional` / `X | None`

Real code is full of "this is a `str`, *or* it might be `None`" - a lookup that can fail, a default that
isn't set yet. Spell that as `X | None` (Python 3.10+): "an `X`, or `None`."

```python runnable
def find_user(user_id: int) -> str | None:    # returns a name, or None if not found
    users = {1: "Ana", 2: "Bo"}
    return users.get(user_id)                  # .get returns None when the key is absent

result = find_user(1)
if result is not None:                         # narrow it: now it's definitely a str
    print(result.upper())
else:
    print("not found")
```
*What just happened:* `str | None` says the return is *either* a string *or* `None`. Because of that, mypy
won't let you blindly call `result.upper()` - `None` has no `.upper()`. The `if result is not None:` check
**narrows** the type: inside that branch, mypy knows `result` is a real `str`, so `.upper()` is safe. This
is typing's everyday payoff - it forces you to handle the "what if it's missing?" case you'd otherwise
forget.

📝 **`Optional[X]`** - the older spelling of the same thing: `Optional[str]` means exactly `str | None`.
You'll see it in existing code and from `from typing import Optional`; on 3.10+, `str | None` reads more
plainly. **`Union[X, Y]`** is the general "one of several types" - `Union[int, str]` is now `int | str`.

So the modern shorthand, all on the `|` operator:

- `str | None` - a string, or nothing (the old `Optional[str]`).
- `int | str` - an int *or* a string (the old `Union[int, str]`).

## A glimpse of `Protocol` - duck typing, but checked

You met **duck typing** earlier: Python doesn't care what *class* an object is, only whether it has the
method you call ("if it quacks, it's a duck"). That's powerful, but a plain type hint can't express "I
accept anything with a `.read()` method" - it only knows concrete class names.

`typing.Protocol` closes that gap. You define a Protocol listing the methods/attributes you need, and any
object that *structurally* has them counts as a match - no inheritance required.

```python runnable
from typing import Protocol

class Readable(Protocol):
    def read(self) -> str: ...      # "anything with a .read() returning str"

def show(source: Readable) -> None:
    print(source.read())

class File:                          # never mentions Readable, but has read()
    def read(self) -> str:
        return "file contents"

show(File())                         # accepted: File structurally matches Readable
```
*What just happened:* `Readable` is a Protocol describing a *shape* - "has a `read()` that returns `str`."
`File` never inherits from `Readable` and never even imports it, yet it satisfies the protocol because it
has the right method. mypy checks that match for you: the flexibility of *if it quacks*, with a tool
verifying the quack before you ship.

(`...` here is a real Python value - the ellipsis literal - used as a stand-in body meaning "no
implementation, just the signature." It runs fine.)

## Running mypy - where the hints earn their keep

All of the above is just annotation until you point a checker at it. **mypy** is the most common one:
install it, run it on your file, and it reports type mismatches without executing your code.

Say you have this file, `account.py`:

```python
def withdraw(balance: int, amount: int) -> int:
    return balance - amount

withdraw(100, "20")     # oops - passing a str where an int is expected
```

This file *runs* and then crashes at runtime with a `TypeError` on the subtraction, since you can't
subtract a string from an int. But mypy catches it *before* you run anything:

```console
$ mypy account.py
account.py:4: error: Argument 2 to "withdraw" has incompatible type "str"; expected "int"  [arg-type]
Found 1 error in 1 file (checked 1 source file)
```
*What just happened:* mypy read your annotations, saw that `withdraw` wants an `int` for `amount`, noticed
you passed `"20"` (a `str`), and pointed at the exact line and argument - `account.py:4`, argument 2. You
fixed a bug by *reading*, not by running and waiting for a crash in production.

⚠️ **Gotcha - hints are NOT enforced at runtime.** This surprises everyone. The annotation `amount: int`
does **not** make Python reject a string at runtime. If you never run mypy, `withdraw(100, "20")` happily
executes until the subtraction blows up later, possibly far from where the wrong value entered. Hints are
*checked* only by a tool you choose to run (mypy, or your IDE doing the same in the background) - Python
itself trusts you completely. A type hint without a checker is a polite comment, nothing more.

```python runnable
def withdraw(balance: int, amount: int) -> int:
    return balance - amount

# Python does NOT stop this - the hint is ignored at runtime:
print(withdraw(100, 20))            # 80 - fine
print(type(withdraw))               # the function still exists, hints and all
```
*What just happened:* the correctly-typed call returned `80`. Python read the annotations, stored them, and
otherwise ignored them - nothing about `int` was enforced as the program ran. The interpreter doesn't
police types; the checker does.

## When typing earns its keep (and when it doesn't)

Type hints aren't free - they're more to write and to read. They pay off most when:

- **The codebase is large or long-lived.** More files and more people means a function's contract needs
  to be machine-checkable instead of remembered.
- **You're writing a library** others import. Hints become documentation *and* give your users' editors
  autocomplete and warnings for free.
- **You want editor superpowers.** With hints, your IDE knows `user.` should offer `.name` and `.email`,
  and warns you the instant you misspell or misuse something - no running required.

They earn their keep less in a 20-line throwaway script or an exploratory notebook, where the ceremony
costs more than it saves. That's the point of *gradual* typing: add them where the safety is worth it,
skip them where it isn't. A common pattern is to start untyped, then add hints to the parts that got big
or scary.

## Recap

1. **Hints change nothing at runtime.** They're for humans, editors, and a type checker - the
   interpreter ignores them.
2. **Basic syntax:** `def greet(name: str) -> str:` for parameters and returns; `count: int = 3` for
   variables.
3. **Generics** say what's inside a container: `list[int]`, `dict[str, int]` (3.9+ lowercase forms).
4. **Maybe-missing values:** `str | None` (old: `Optional[str]`); **either/or:** `int | str` (old:
   `Union[int, str]`), both on the `|` operator in 3.10+.
5. **`Protocol`** is duck typing, checked - match by *shape* (the methods present), not by inheritance.
6. **mypy** reads your hints and flags mismatches *before* you run, pointing at the exact line. But hints
   are **not** enforced at runtime - without a checker, a wrong type still runs and may blow up later.
7. Typing earns its keep in **bigger codebases, libraries, and for editor autocomplete**; skip it for
   throwaway scripts.

You can now annotate code and let a tool catch a whole class of bugs early. Next, hints go to work for
*modeling*: dataclasses turn "a bag of typed fields" into a clean, declarative class with almost no
boilerplate.

## Quick check

Test the one sticky idea - hints don't run, they're *checked*. Pick the best answer for each.

```quiz
[
  {
    "q": "You write `def f(x: int) -> int: return x` and then call `f(\"hello\")`. You never run mypy. What does Python do at runtime?",
    "choices": [
      "Raises a TypeError immediately because \"hello\" isn't an int",
      "Runs the call normally - the hint is ignored at runtime",
      "Refuses to define the function until the type is fixed",
      "Silently converts \"hello\" to an int"
    ],
    "answer": 1,
    "explain": "Type hints change nothing at runtime. Python stores annotations but never enforces them. The call runs; only a checker like mypy would have flagged the mismatch beforehand."
  },
  {
    "q": "Which tool actually catches the mismatch between a hint and the value you passed?",
    "choices": [
      "The Python interpreter, when it runs the line",
      "A type checker like mypy (or your IDE doing the same check)",
      "The `__annotations__` dictionary, automatically",
      "Nothing - hints are purely decorative comments"
    ],
    "answer": 1,
    "explain": "mypy reads your annotations and reports mismatches before you run anything. The interpreter just runs; it doesn't police types. Hints aren't merely decorative - they have teeth, but only when a checker reads them."
  },
  {
    "q": "What does the annotation `str | None` mean?",
    "choices": [
      "A string that will be converted to None",
      "Either a string or None - i.e. the value is optional",
      "A string that must never be None",
      "Two separate variables, a str and a None"
    ],
    "answer": 1,
    "explain": "`str | None` (the modern spelling of `Optional[str]`) means the value is either a `str` or `None`. It's how you say \"this might be missing,\" forcing you to handle the None case."
  }
]
```


---

# Dataclasses & Modern Modeling

A huge fraction of the classes you'll ever write are just *bags of fields*. A `Point` with an `x` and a
`y`. A `User` with a name, an email, and an age. There's no clever behavior - the class exists only to
hold a few values together and let you compare and print them.

Back in [Phase 6](06-objects-and-classes.md) you wrote these by hand, and in
[Phase 10](10-the-data-model.md) you learned the comparing and printing live in dunder methods like
`__repr__` and `__eq__`. Put those two facts together and a problem appears: for every bag-of-fields
class, you write the *same* `__init__`, `__repr__`, and `__eq__` by hand, every time - tedious, easy to
get subtly wrong, pure ceremony.

Modern Python kills that ceremony. This phase covers the tools that write the boilerplate for you so you
can describe *what your data is* and stop typing the parts a machine could.

## The boilerplate problem - see it first

Here's a plain, hand-written bag-of-fields class. Nothing here is clever - every line is obligatory.

```python
class Point:
    def __init__(self, x, y):
        self.x = x
        self.y = y

    def __repr__(self):
        return f"Point(x={self.x!r}, y={self.y!r})"

    def __eq__(self, other):
        if not isinstance(other, Point):
            return NotImplemented
        return (self.x, self.y) == (other.x, other.y)
```

Three methods, and none says anything about what a `Point` *means*. The only real information in the
whole class is "a point has an `x` and a `y`." Everything else is machinery you're forced to hand-crank:
store the args, print nicely, make equal coordinates compare equal.

⚠️ It's not just tedious - it's a place bugs hide. Add a `z` coordinate later and you must remember to
touch `__init__`, `__repr__`, *and* `__eq__`. Miss one and you get a class that constructs fine but
prints the old shape or compares wrong, quietly.

## `@dataclass` - describe the fields, get the methods for free

**What it actually is.** `@dataclass` is a **decorator** from the standard library that reads the typed
fields you declare at the top of a class and *generates* `__init__`, `__repr__`, and `__eq__` from them.
You write the data; it writes the ceremony.

📝 **Decorator** - a thing you put above a class or function with `@name` that transforms it. Here,
`@dataclass` takes your field declarations and adds the dunder methods before you ever use the class.

It leans directly on the [type hints](14-type-hints.md) from the last phase: the `x: int` annotations
aren't decoration here, they're how the dataclass knows what the fields *are*. Watch the same `Point`
collapse:

```python runnable
from dataclasses import dataclass

@dataclass
class Point:
    x: int
    y: int

p = Point(3, 4)
print(p)            # the generated __repr__
print(p == Point(3, 4))   # the generated __eq__
```
```console
$ python points.py
Point(x=3, y=4)
True
```
*What just happened:* The two lines `x: int` and `y: int` are the entire class now. `@dataclass` saw them
and wrote an `__init__(self, x, y)` that stores both, a `__repr__` that prints `Point(x=3, y=4)`, and an
`__eq__` that compares by field values - why two separately-built points with the same coordinates came
back `True`. That matters because a plain class compares by *identity* (are these the literal same
object?) by default, so `Point(3, 4) == Point(3, 4)` would be `False` without the generated `__eq__`.

> 💡 **Key point.** A dataclass isn't a new kind of object - it's a *normal class* with the boring dunders
> filled in for you. You can still add methods, properties, and anything else a regular class has.

## Defaults - and the mutable-default trap, again

Fields can have defaults, so callers can omit them. You write them like default arguments:

```python runnable
from dataclasses import dataclass

@dataclass
class User:
    name: str
    active: bool = True      # default: callers can leave it off

print(User("Ana"))
print(User("Bo", active=False))
```
```console
$ python users.py
User(name='Ana', active=True)
User(name='Bo', active=False)
```
*What just happened:* `active: bool = True` gave the field a fallback, so `User("Ana")` filled it in
automatically while `User("Bo", active=False)` overrode it - it reads exactly like default arguments in a
normal `__init__`, because that's what the dataclass generates.

But the moment a default is a **mutable** value - a list, a dict, a set - you're standing on the same
landmine from [Phase 4](04-control-flow-and-functions.md): mutable defaults are created *once* and shared.
Python refuses to let you do it the naive way:

```console
$ python tags.py
ValueError: mutable default <class 'list'> for field tags is not allowed: use default_factory
```
*What just happened:* You wrote `tags: list = []` and the dataclass machinery stopped you cold. A single
`[]` written at class-definition time would be shared by *every* `User` instance - append a tag to one
user and it'd appear on all of them. Rather than hand you that footgun, dataclasses raise a `ValueError`.

**The fix - `field(default_factory=...)`.** Instead of giving a default *value*, you give a default
*recipe*: a zero-argument callable that the dataclass calls fresh for each new instance.

```python runnable
from dataclasses import dataclass, field

@dataclass
class User:
    name: str
    tags: list = field(default_factory=list)   # a fresh [] per instance

a = User("Ana")
b = User("Bo")
a.tags.append("admin")     # only touches a's list
print(a)
print(b)
```
```console
$ python tags.py
User(name='Ana', tags=['admin'])
User(name='Bo', tags=[])
```
*What just happened:* `default_factory=list` told the dataclass "when a `User` is built without `tags`,
call `list()` to make a brand-new empty list for it." So `a` and `b` got *separate* lists - appending to
`a.tags` left `b.tags` empty. Same trap, same cure as function arguments, just spelled with `field()`. Any
zero-arg callable works: `default_factory=dict`, `default_factory=set`, or your own `lambda: [0, 0, 0]`.

⚠️ **The rule to remember:** immutable default (number, string, `True`, `None`) → write it plainly
(`active: bool = True`). Mutable default (list, dict, set) → always `field(default_factory=...)`. The
dataclass enforces this for you, which is one of the nicer things it does.

## `frozen=True` - make instances immutable and hashable

By default a dataclass is mutable: you can reassign `p.x = 99` after the fact. Sometimes that's exactly
wrong. A `Point`, a `Color`, a `Coordinate` - these are *values*, and a value that can be quietly mutated
under you is a source of bugs. You want it to behave like a number, fixed once created.

**What it does.** Passing `frozen=True` to the decorator makes assignment to fields raise an error after
construction, and - because the object can no longer change - Python can also generate a `__hash__`,
which means frozen instances can go into sets and be used as dict keys.

```python runnable
from dataclasses import dataclass

@dataclass(frozen=True)
class Point:
    x: int
    y: int

p = Point(1, 2)
print(p)
seen = {p, Point(1, 2), Point(3, 4)}   # usable in a set now
print(len(seen))
p.x = 99                                # this is no longer allowed
```
```console
$ python frozen.py
Point(x=1, y=2)
2
dataclasses.FrozenInstanceError: cannot assign to field 'x'
```
*What just happened:* `frozen=True` locked the instance. Two of the three points in the set were equal
(`Point(1, 2)` twice), so the set collapsed them and `len` is `2` - that only works because frozen
dataclasses are **hashable**. Then `p.x = 99` raised `FrozenInstanceError`: the object refuses to be
mutated after birth. A bag of fields is now a proper immutable value.

📝 **Hashable** - an object Python can reduce to a fixed number so it can live in a `set` or be a `dict`
key. Mutable objects (like lists, or default dataclasses) generally aren't, because if they changed, the
number would lie. Freezing makes the object unchanging, so hashing becomes safe.

Here's the small decision in one picture:

```mermaid
flowchart TD
  A{Will it ever change<br/>after creation?} -->|Yes| B[plain @dataclass]
  A -->|No| C["@dataclass(frozen=True)"]
  C --> D[immutable + hashable:<br/>safe in sets and as dict keys]
```

## `typing.NamedTuple` - the lightweight immutable alternative

There's an even lighter option for the immutable case, predating the modern dataclass: a typed named
tuple. It's a tuple under the hood - immutable and hashable for free - but with *named* fields instead of
`[0]`/`[1]` positions.

**What it actually is.** `typing.NamedTuple` lets you declare a tuple subclass with typed, named fields.
You get the `__repr__` and `__eq__` for free like a dataclass, plus everything tuples already do:
indexing, unpacking, immutability.

```python runnable
from typing import NamedTuple

class Point(NamedTuple):
    x: int
    y: int

p = Point(3, 4)
print(p)
print(p.x, p[0])        # access by name OR by position
a, b = p                # it unpacks like the tuple it is
print(a, b)
```
```console
$ python named.py
Point(x=3, y=4)
3 3
3 4
```
*What just happened:* `Point(3, 4)` built a tuple whose two slots also answer to `.x` and `.y`. So `p.x`
and `p[0]` are the same value, and `a, b = p` unpacks it exactly like `(3, 4)` because it *is* a tuple -
immutable and hashable automatically, with none of the `frozen=True` ceremony.

**When to reach for which.** Both model immutable bags of fields. The plain trade-off:

| | `typing.NamedTuple` | `@dataclass(frozen=True)` |
|---|---|---|
| Immutable & hashable | yes, automatically | yes, with `frozen=True` |
| Acts like a tuple (index, unpack) | yes - sometimes handy, sometimes a footgun | no, it's its own type |
| Mutable option | no | yes, drop `frozen=True` |
| Defaults, `field()`, rich config | limited | full |

Reach for `NamedTuple` for a small, immutable record that's pleasant to unpack and pass around. Reach for
`@dataclass` when you want more control - mutability when needed, `default_factory`, room to grow methods
onto the class. (Judgment, not law: most teams default to `@dataclass` and use `NamedTuple` for genuinely
tuple-shaped cases.)

## A one-paragraph nod to pydantic

Everything above *trusts its inputs*. A dataclass `User(age="not a number")` happily stores the string -
the `: int` hint is a note for humans and type-checkers ([Phase 14](14-type-hints.md)), not a runtime
guard. That's fine inside your own code, but at a *boundary* - JSON from an API, a config file, a form
submission - you want the data **validated and coerced** before it gets in. That's
[**pydantic**](https://docs.pydantic.dev/), a third-party library (`pip install pydantic`; not in the
standard library, which is why the snippet below isn't runnable here). It looks almost identical to a
dataclass, but *enforces* types at construction time and raises a clear error when the data is wrong:

```python
from pydantic import BaseModel

class User(BaseModel):
    name: str
    age: int

User(name="Ana", age="30")    # the string "30" is coerced to int 30
User(name="Bo", age="oops")   # raises ValidationError - not a valid integer
```

Rule of thumb: **dataclasses for data you already trust** (objects you build inside your own program);
**pydantic at the edges** where untrusted data crosses into your system. You'll meet it for real when you
build APIs.

## Recap

1. Bag-of-fields classes force you to hand-write `__init__`, `__repr__`, and `__eq__` - pure boilerplate,
   and a place bugs hide when the class grows.
2. **`@dataclass`** generates those three from your typed fields. It's a normal class with the dunders
   filled in; you can still add methods.
3. **Defaults** are written like default arguments (`active: bool = True`) - but a **mutable** default is
   forbidden; use **`field(default_factory=list)`** so each instance gets its own fresh list/dict/set.
   Same trap, same cure as [Phase 4](04-control-flow-and-functions.md).
4. **`frozen=True`** makes instances immutable *and* hashable - safe as values, in sets, and as dict keys.
5. **`typing.NamedTuple`** is the lightweight immutable alternative: tuple-backed, named fields, free
   immutability and hashing.
6. **pydantic** (third-party) validates and coerces at the boundary - reach for it where untrusted data
   enters, and keep dataclasses for data you already trust.

You can now describe your data cleanly and let the language write the ceremony. Next, leaving the world
of single-threaded code to ask the question that confuses everyone: why don't Python threads make CPU
work faster?

## Quick check

Before you move on, three questions to make sure the dataclass essentials stuck - what the decorator generates, the mutable-default fix, and what `frozen=True` buys you.

```quiz
[
  {
    "q": "You write a class with two typed fields and put @dataclass above it. Which methods does the decorator generate for you by default?",
    "choices": [
      "__init__, __repr__, and __eq__",
      "Only __init__",
      "__hash__ and __slots__",
      "None - @dataclass just documents the fields for type-checkers"
    ],
    "answer": 0,
    "explain": "@dataclass reads your typed fields and generates __init__ (stores the args), __repr__ (prints Point(x=3, y=4)), and __eq__ (compares by field values) from them. __hash__ is only added when you also pass frozen=True."
  },
  {
    "q": "You write `tags: list = []` in a dataclass and Python raises a ValueError. What's the correct fix?",
    "choices": [
      "tags: list = field(default_factory=list)",
      "tags: list = field(default=[])",
      "tags: list = None, then build the list in __post_init__",
      "Add frozen=True so the shared list can't be mutated"
    ],
    "answer": 0,
    "explain": "A single [] written at class-definition time would be shared by every instance. field(default_factory=list) gives a recipe Python calls fresh for each new instance, so each one gets its own empty list - the same trap and cure as mutable default arguments."
  },
  {
    "q": "What does passing frozen=True to @dataclass give you?",
    "choices": [
      "Instances become immutable and hashable - assignment after construction raises, and they can go in sets and be dict keys",
      "It makes every field optional with a default of None",
      "It speeds up attribute access by freezing the class layout",
      "It validates and coerces field types at construction time"
    ],
    "answer": 0,
    "explain": "frozen=True makes assignment to fields raise FrozenInstanceError after construction. Because the object can no longer change, Python can also generate __hash__, so frozen instances are hashable - usable in sets and as dict keys. (Type validation/coercion is pydantic's job, not frozen's.)"
  }
]
```


---

# Concurrency & the GIL

This is the corner of Python that gets the most confident-sounding wrong answers. You'll hear "Python
can't do threads," "the GIL makes Python slow," "use async for everything." Each is a true thing mangled
into a false one. The reality is cleaner than the folklore, and once you have the right mental model
you'll know exactly which tool to reach for - and why the one you tried first didn't help.

There are three tools. The whole phase is about telling them apart and pointing each at the job it's
actually good at.

## The one distinction everything rests on: I/O-bound vs CPU-bound

Before any tool, install this idea, because the right choice falls straight out of it.

**What it actually is.** Every slow piece of work is slow for one of two reasons:

- **I/O-bound** - your program is *waiting*: on a web response, a database query, a file off disk, a
  network socket. The CPU is mostly idle; the clock just ticks while something else takes its time.
- **CPU-bound** - your program is *computing*: crunching numbers, resizing images, parsing a huge blob,
  hashing. The CPU is pinned at 100%; there's no waiting to overlap.

📝 **I/O-bound / CPU-bound** - "bound" means "the thing limiting your speed." If faster I/O would help,
you're I/O-bound. If a faster CPU would help, you're CPU-bound.

> 💡 **Key point.** Almost every concurrency decision in Python is just answering one question: *is this
> work waiting, or computing?* Hold that question. The GIL, the three tools, the gotcha - all of it
> reduces to which side of this line your work sits on.

## The three tools, and the problem each one solves

Python hands you three different mechanisms, and they are *not* interchangeable - they solve different
problems.

📝 **Concurrency** - making progress on several things in overlapping time spans (juggling). **Parallelism**
- running several things at the same instant on different CPU cores (many hands). Not the same word, and
the difference is the heart of this phase.

- **`threading`** - multiple threads inside *one* process, taking turns. Great for overlapping lots of
  *waiting*. This is concurrency, not (in CPython) true parallelism - the GIL below is why.
- **`asyncio`** - a single thread that juggles thousands of waiting tasks using `async`/`await`, switching
  whenever one would block. Also for I/O, also concurrency, but cooperative and very lightweight. Covered
  in depth in [Async/Await & the Event Loop](/guides/async-await-and-the-event-loop).
- **`multiprocessing`** - multiple *separate* Python processes, each with its own interpreter and memory.
  Real parallelism: four processes can pin four cores at once. The tool for CPU-bound work.

That's the menu. You can't just pick the first one for everything, because of the GIL.

## The GIL, explained straight

This is usually either hand-waved or turned into a boogeyman. Here's the straight version.

📝 **GIL - Global Interpreter Lock.** A single lock inside CPython (the standard Python you almost
certainly run). The rule it enforces: **only one thread can execute Python bytecode at a time.** A thread
must hold the GIL to run your Python code, and there's exactly one GIL per process.

**Why it exists.** It's a trade-off, not malice or laziness. The GIL makes CPython's memory management
simple and fast for the common single-threaded case, and lets C extensions be written without worrying
about thread-safety at every turn. The cost of that simplicity is what everyone trips over next.

**What it does in real life - the plain two-part truth:**

1. **For CPU-bound work, threads do NOT speed you up.** Spin up four threads to crunch numbers and they'll
   take turns holding the one GIL - only one runs Python at any instant. Four threads finish in *about the
   same wall-clock time* as one, sometimes slightly worse from switching overhead. No free cores were
   used. This is the single most surprising fact about Python threading, and it's true.

2. **For I/O-bound work, threads DO help - a lot.** The saving grace: **a thread releases the GIL while it
   waits on I/O.** When a thread blocks on a network read or disk fetch, it hands the GIL to another
   thread, which runs while the first waits - the waiting overlaps. So ten threads downloading ten URLs
   really do progress together, not because they compute in parallel but because mostly they're all
   *waiting* in parallel, which the GIL happily allows.

The same release happens inside well-written C extensions: NumPy drops the GIL during heavy array math, so
threaded NumPy code *can* use multiple cores. For *your* plain-Python loop, assume the GIL is held the
whole time.

Here's the shape of it: threads share one interpreter and one GIL; processes each get their own.

```mermaid
flowchart TD
  subgraph Process["One process - threading"]
    GIL{{one GIL}}
    T1[Thread 1] --> GIL
    T2[Thread 2] --> GIL
    T3[Thread 3] --> GIL
  end
  subgraph Procs["multiprocessing"]
    P1[Process A · own interpreter]
    P2[Process B · own interpreter]
  end
```

*One idea:* on the left, three threads fight over a single GIL - only one runs Python at a time. On the
right, each process carries its *own* interpreter and GIL, so they genuinely run at the same instant on
different cores. That picture is the whole decision.

## The decision rule

You almost never have to agonize over this. It collapses to one line:

> **I/O-bound → `threading` or `asyncio`. CPU-bound → `multiprocessing`.**

- **Waiting on the world** (HTTP calls, DB queries, files, sockets)? Use **threads** (simplest) or
  **asyncio** (when you have *thousands* of concurrent waits and want them lightweight). The GIL is
  released during the waits, so concurrency is real.
- **Burning CPU** (math, parsing, image work, compression)? Use **`multiprocessing`**. Separate processes
  sidestep the GIL entirely, giving true parallelism across cores. The cost: processes don't share memory,
  so data crossing between them gets *pickled* (serialized) and copied - not worth it without real CPU
  work to justify the overhead.

📝 **Pickling** - Python's built-in serialization. To send an object to another process, it's converted to
bytes and rebuilt on the other side. Cheap for small data, not free for large data.

## What each one looks like

You won't run these here - threads, processes, and event loops don't behave deterministically in a
browser sandbox, so treat these as illustrative. They're faithful to a real terminal.

### asyncio - one thread juggling waits

```python
import asyncio

async def fetch(name, seconds):
    print(f"{name} starting")
    await asyncio.sleep(seconds)        # stand-in for a network wait; yields control here
    print(f"{name} done")
    return name

async def main():
    # run three "fetches" concurrently, not one-after-another
    results = await asyncio.gather(
        fetch("A", 2),
        fetch("B", 1),
        fetch("C", 3),
    )
    print("all done:", results)

asyncio.run(main())
```
```console
$ python fetch.py
A starting
B starting
C starting
B done
A done
C done
all done: ['A', 'B', 'C']
```
*What just happened:* all three started immediately. At each `await asyncio.sleep(...)`, the task said
"I'm about to wait - someone else go ahead," and the single event loop switched to another task. So `B`
(1s) finished before `A` (2s) before `C` (3s), and the whole thing took about 3 seconds, not 2+1+3 = 6.
One thread, no GIL fight, thousands of these possible - the I/O-bound tool when you have *many* waits. The
full machinery - the event loop, what `await` really does - is in
[Async/Await & the Event Loop](/guides/async-await-and-the-event-loop).

### multiprocessing - real parallelism for CPU work

```python
from multiprocessing import Pool

def heavy(n):                           # pure CPU: no waiting, just computing
    return sum(i * i for i in range(n))

if __name__ == "__main__":              # required on Windows/macOS - see gotcha below
    with Pool(4) as pool:               # four worker processes, four cores
        results = pool.map(heavy, [10_000_000] * 4)
    print(results)
```
```console
$ python crunch.py
[333333283333335000000, 333333283333335000000, 333333283333335000000, 333333283333335000000]
```
*What just happened:* `Pool(4)` started four separate Python processes, each with its own interpreter and
GIL. `pool.map` handed one `heavy` call to each, and they ran *genuinely simultaneously* on four cores.
The four results came back and got copied (unpickled) into your main process - the speedup threads
couldn't give you, because the GIL is per-process and now there are four of them.

⚠️ **Gotcha - `if __name__ == "__main__":` is mandatory here.** On Windows and macOS, child processes
*re-import your script* to start up. Without that guard, each child re-runs the `Pool(...)` line, spawning
more children, which re-import and spawn again - an explosion. The guard makes the spawning code run only
in the original process; leave it off and `multiprocessing` misbehaves.

## ⚠️ The gotcha that sends everyone here in the first place

This is the one that drags people here, usually confused and a little annoyed:

> You have a slow number-crunching loop. You think "I'll just throw it on a few threads." You do. You
> measure. It's **exactly as slow as before** - maybe slower. Nothing went parallel.

That's not a bug in your code. **That's the GIL.** Your work was CPU-bound, so the four threads spent the
whole time taking turns holding the one lock - only ever one running Python at a time. Threads were never
going to help, because nothing was *waiting*; it was all *computing*.

The fix is the decision rule: CPU-bound work wants **`multiprocessing`**, not threads. Swap the thread pool
for a process pool and the same loop spreads across your cores and actually gets faster.

🪖 **War story.** Nearly every Python developer writes this exact bug once: a slow image-processing or
data-crunching script, "speed it up with threads," no change, a half-hour staring at the profiler before
the GIL clicks into place. Knowing it ahead of time is the whole reason this phase exists.

## One straight note on the future

The GIL has been CPython's defining constraint for decades, and that's finally starting to shift:
**free-threaded ("no-GIL") CPython is real** - an optional build (experimental in Python 3.13,
officially supported since 3.14) where the GIL can be disabled, letting threads run Python in genuine
parallel. It's still not the default, and many C extensions aren't ready for it - so for everything you
write today, plan around the GIL exactly as described above. The ground may move under this in the next
few years.

## Recap

1. Every slow task is either **I/O-bound** (waiting) or **CPU-bound** (computing). That single question
   drives every decision here.
2. Three tools, three jobs: **`threading`** (overlap waiting), **`asyncio`** (overlap *lots* of waiting,
   lightweight), **`multiprocessing`** (real parallelism for computing).
3. The **GIL** lets only one thread execute Python bytecode at a time per process. So threads **don't**
   speed up CPU-bound work - but the GIL is **released during I/O** (and by C extensions like NumPy), so
   threads **do** speed up I/O-bound work.
4. The rule: **I/O-bound → threads or asyncio; CPU-bound → multiprocessing** (separate processes = separate
   GILs = true parallelism, at the cost of copying data between them).
5. The classic trap - threading a number-crunching loop and seeing no speedup - *is* the GIL telling you
   that you wanted `multiprocessing`.
6. **Free-threaded / no-GIL CPython is real** (officially supported since Python 3.14) but not the
   default; design around the GIL for now.

Concurrency is one half of "making Python fast." The other half is the work each core actually does - how
Python uses memory, where the time goes, and how to measure it instead of guessing. That's next.

## Quick check

The GIL is the one idea that has to stick. Test yourself before moving on.

```quiz
[
  {
    "q": "You have a CPU-bound loop (pure number crunching, no waiting). You split it across four threads and measure: it's no faster than one thread. Why?",
    "choices": [
      "Your CPU only has one core",
      "The GIL lets only one thread execute Python bytecode at a time, so the four threads just take turns - no parallelism",
      "Threads in Python are always slower than a single thread",
      "You forgot to call thread.start() on the extra threads"
    ],
    "answer": 1,
    "explain": "That's the GIL doing exactly what it does: one thread runs Python bytecode at a time per process. With nothing waiting, the threads serialize on the lock and you get zero speedup (sometimes a little worse from switching overhead)."
  },
  {
    "q": "You need to download 200 URLs, each mostly spent waiting on the network. Which tool fits best?",
    "choices": [
      "multiprocessing - you need true parallelism",
      "threading or asyncio - the work is I/O-bound, and the GIL is released while a thread waits",
      "Nothing helps; the GIL blocks all concurrency",
      "A single plain loop - concurrency can't help with network calls"
    ],
    "answer": 1,
    "explain": "This is I/O-bound: the program is waiting, not computing. A thread releases the GIL while it waits, so the waits overlap. threading (simplest) or asyncio (when you have thousands of lightweight waits) is the right call - not multiprocessing."
  },
  {
    "q": "You have a genuinely CPU-bound job (resizing thousands of images in pure Python) and want it to actually use all your cores. What's the right tool?",
    "choices": [
      "threading - spin up one thread per core",
      "asyncio - async/await makes everything parallel",
      "multiprocessing - separate processes each get their own interpreter and GIL, giving true parallelism across cores",
      "It's impossible to use multiple cores from Python"
    ],
    "answer": 2,
    "explain": "CPU-bound work wants multiprocessing. Each process has its own GIL, so they run genuinely simultaneously on different cores - at the cost of pickling/copying data between them. Threads and asyncio only help when work is waiting, not computing."
  }
]
```


---

# Performance & Memory

At some point someone tells you "Python is slow," and it lands as a vague accusation you can't quite
defend against. Slow compared to what? Slow at *what*? And when your own code really is too slow, the
panic is worse - you start rewriting random pieces by feel, hoping something helps, and usually it
doesn't. You burn an afternoon and the program runs the same speed it did before lunch.

Both problems come from the same gap: you don't have a mental model of *what Python is doing under your
code*, so you can't reason about where the time goes. This phase fixes that: why Python earns the "slow"
label (and why it usually doesn't matter), how CPython actually executes your program and manages memory,
then the rule that saves you from wasted afternoons - **measure, don't guess** - and the speedups that
genuinely work, in the order worth trying them.

## Why Python is "slow"

**What's actually going on.** Python is **interpreted** and **dynamically typed**, and those two facts
are the whole story. Compare a quick `a + b`:

- In a compiled, statically-typed language (C, Rust), the compiler knew the types ahead of time. `a + b`
  becomes one machine instruction - add these two integers in these two registers. Done.
- In Python, nothing is known ahead of time. At the moment `a + b` runs, the interpreter asks: what *is*
  `a`? What *is* `b`? Does this kind of object know how to be added? It looks up the types, finds the
  right `__add__` method, calls it, and wraps the result back into a Python object on the heap.

That lookup-and-dispatch dance happens for *every single operation*, every time it runs. The flexibility
you love - duck typing, swapping a list for a generator, monkey-patching in a test - is bought with
runtime work a compiled language did once, up front.

📝 **Interpreted** - your source is executed by another program (the interpreter) at runtime, rather than
being translated to machine code ahead of time. **Dynamically typed** - a variable's type is discovered
while the program runs, not declared in the source.

> 💡 **Key point.** Python isn't slow because it was written badly. It's slow at raw per-operation work
> *by design*, as the price of being dynamic and flexible. That trade only matters in the small fraction
> of code that does heavy per-operation work in a tight loop - and that's exactly the part you can hand
> off to fast C.

**Why this usually doesn't matter.** Most programs spend their time *waiting* - on the network, the disk,
the database - not on CPU work. When you're waiting on a web response, it makes no difference that the
language is interpreted; you'd be waiting in C too. Python is "slow" precisely where it's rarely the
bottleneck, and fast enough everywhere else. Keep this phrase in your head: slow *language*, fast
*ecosystem* - the heavy lifting lives in C libraries (more below).

## How CPython runs your code

📝 **CPython** - the standard, reference implementation of Python, the one you get from python.org and
almost certainly the one you're running. "Python the language" and "CPython the program that runs it" are
worth keeping separate; other implementations (PyPy, etc.) make different speed trade-offs.

**What actually happens when you run a `.py` file.** It's two steps, not one. CPython first *compiles*
your source into **bytecode** - a compact list of simple instructions for a virtual machine. Then the
**eval loop** walks that bytecode one instruction at a time and does what each one says.

```mermaid
flowchart LR
  src[your .py source] -->|compile| bc[bytecode]
  bc -->|eval loop runs each op| result[result]
```

📝 **Bytecode** - not machine code your CPU runs directly, but instructions for CPython's own virtual
machine: `LOAD_FAST`, `BINARY_OP`, `CALL`, `RETURN_VALUE`, and friends. The `.pyc` files in your
`__pycache__` folder are cached bytecode, so a module that hasn't changed skips re-compiling next time.

**Seeing it for real.** The `dis` module ("disassemble") shows the bytecode for any function. You'll
rarely need this, but looking once demystifies the whole thing.

```python
import dis

def add(a, b):
    return a + b

dis.dis(add)
```
```console
$ python show_bytecode.py
  3           RESUME                   0

  4           LOAD_FAST                a
              LOAD_FAST                b
              BINARY_OP                0 (+)
              RETURN_VALUE
```
*What just happened:* `dis` printed the instructions CPython runs for `add`: load `a`, load `b`, perform
`+`, return the top of the stack. Each step is the eval loop doing a bit of work, and `BINARY_OP` is where
that runtime type lookup from the last section actually lives. (Opcode names shift between Python
versions, so don't memorize them; the shape is what matters.)

**Why this is worth knowing.** Once you can picture "source → bytecode → an eval loop grinding through
ops," two things stop being mysterious: why pushing work into a single C-implemented built-in beats a
hand-written loop (one C call versus thousands of trips through the eval loop), and why micro-rewrites of
pure-Python loops rarely help much (you're still grinding the same loop, just with slightly different ops).

## Memory - how objects get freed

You never call `free()` in Python. Memory is managed for you, and it's worth knowing how - it explains
both why Python is convenient and where it can surprise you.

**The main mechanism: reference counting.** Every Python object carries a count of how many things
currently refer to it. Bind it to a new name, the count goes up. A name goes out of scope or gets
reassigned, the count goes down. The instant that count hits **zero** - nothing refers to this object
anymore - CPython frees it immediately.

```mermaid
flowchart LR
  a["x = []  (refs: 1)"] --> b["y = x  (refs: 2)"]
  b --> c["del x  (refs: 1)"]
  c --> d["y = None  (refs: 0)"]
  d --> e[freed immediately]
```

*One idea:* the object lives exactly as long as something points at it. When the last reference drops, it
goes away right then - no waiting, no scheduled sweep.

**The backup mechanism: a cyclic garbage collector.** Reference counting has one blind spot: a **cycle**.
If object A refers to B and B refers back to A, their counts never reach zero even after *you* let go of
both - each props up the other. So CPython also runs a periodic **garbage collector** whose only job is
finding these unreachable cycles and cleaning them up.

📝 **Reference cycle** - two or more objects that refer to each other, so reference counting alone can't
free them. The cyclic GC exists specifically to catch these.

> 📝 This is a deliberately short version. The full picture - how the cyclic collector decides what's
> truly unreachable, generational collection, when to care - lives in
> [Memory & Garbage Collection](/guides/memory-and-garbage-collection). Read that when memory behavior
> (not speed) is what's biting you.

**Why this touches performance.** Two practical consequences. First, creating and destroying *millions* of
small objects isn't free - each is a heap allocation plus refcount bookkeeping, part of why a pure-Python
numeric loop over a huge dataset is slow (you're minting a Python object per number). That's the exact
pain NumPy removes, below. Second, the GC's occasional cycle-hunting pauses are usually invisible, but in
latency-sensitive code they're a knob you can tune. Both are reasons to *measure* before assuming you know
where the cost is.

## The cardinal rule: measure, don't guess

Here is the single most important thing in this phase, and the one developers ignore most:

> 💡 **Key point.** You are *bad* at guessing where your program spends its time. Everyone is. The hot
> spot is almost never where intuition points. **Profile first, then optimize the part that's actually
> slow** - and nothing else.

Before you touch a line of code for speed, get the lay of the land from these two companion guides:
[What "Performance" Even Means](/guides/what-performance-means) (latency vs. throughput, what you're even
trying to make faster) and [Profiling 101](/guides/profiling-101) (how to find the hot spot instead of
guessing at it).

**Timing a small piece: `timeit`.** For comparing two ways of doing one small thing, `timeit` runs it many
times and reports how long, handling the fiddly bits (warm-up, repeated runs) for you.

```console
$ python -m timeit -s "data = list(range(1000))" "sum(data)"
50000 loops, best of 5: ... usec per loop
```
*What just happened:* `timeit` ran `sum(data)` tens of thousands of times and reported the best per-loop
time. The `-s` setup string (building the list) runs once and isn't counted. The actual microsecond number
depends entirely on your machine and load - treat any timing you see written down, including in this
guide, as *illustrative*, never a fact about your code. Run it yourself.

**Finding the hot spot in a whole program: `cProfile`.** `timeit` answers "which of these two snippets is
faster." `cProfile` answers the bigger question - "in my actual program, where does the time *go*?" - by
running your code and reporting how long was spent in each function.

```console
$ python -m cProfile -s cumulative my_program.py
         ... function calls in ... seconds

   ncalls  tottime  percall  cumtime  percall filename:lineno(function)
        1    0.001    0.001    2.104    2.104 my_program.py:1(<module>)
        1    0.003    0.003    2.090    2.090 my_program.py:12(process_all)
     5000    1.870    0.000    1.980    0.000 my_program.py:30(parse_line)
        ...
```
*What just happened:* `cProfile` ranked functions by cumulative time. Reading it, `parse_line` is eating
the program - called 5000 times and dominating the total. *That's* where optimization effort belongs;
everything else is noise. Without this you'd be guessing, and probably "optimize" `process_all` because
it's at the top of the file - and gain nothing. (Numbers above are illustrative, showing the *shape* of
the output, not a benchmark.)

⚠️ **Gotcha - premature optimization.** This is the big one, the mistake that wastes more developer time
than almost anything else: rewriting code for speed *before* measuring. Someone reads that "list
comprehensions are fast" or "f-strings beat `.format()`," sprints to apply it everywhere, and rewrites the
wrong 5% from a hunch - code that wasn't slow, often less readable in the bargain - while the actual
bottleneck (a redundant database query, an accidental O(n²) loop) sits untouched. The fix is a discipline,
not a trick: **profile, find the real hot spot, fix only that, measure again to confirm it helped.** Skip
the profile and you're guessing - and you'll usually guess wrong.

## The real speedups, in order

When profiling *has* shown you a genuine hot spot, here's where to reach, top to bottom. Try them in this
order - the early ones win big and cost little; the later ones cost more and matter less often.

**1. A better algorithm or data structure.** This dwarfs everything else. No amount of micro-tuning
rescues an O(n²) approach when an O(n) one exists. The classic: you're checking "is this item in the
collection?" inside a loop, and the collection is a `list`, so each check scans the whole thing. Switch it
to a `set` - membership becomes near-instant - and a slow loop becomes fast, untouched otherwise. This is
the first thing to look at, always, and the cheapest. (Worth a refresher?
[Big-O Without the Math Panic](/guides/big-o-without-the-math-panic) is the backbone here.)

**2. Built-ins and the standard library.** Functions like `sum`, `min`, `max`, `sorted`, `any`, and the
tools in `itertools` and `collections` are implemented in **C**. When one does what your loop does, it
does it in a single fast C call instead of thousands of trips through the eval loop. Prefer `sum(data)`
over a hand-rolled accumulator loop - same result, far less interpreter overhead. Rule of thumb: *if a
built-in already does it, let it.*

**3. Cache repeated work with `functools.lru_cache`.** If a function is **pure** (same inputs → same
output, no side effects) and gets called with the same arguments over and over, have Python remember past
results instead of recomputing them. One decorator does it.

```python runnable
import functools

@functools.lru_cache(maxsize=None)
def fib(n):
    if n < 2:
        return n
    return fib(n - 1) + fib(n - 2)

print(fib(35))
print(fib.cache_info())
```
```console
$ python cache_demo.py
9227465
CacheInfo(hits=33, misses=36, maxsize=None, currsize=36)
```
*What just happened:* `fib` is the naive recursive Fibonacci, which normally recomputes the same values
an exploding number of times. With `@lru_cache`, each `fib(n)` is computed *once* and stored; every later
call with that same `n` is a **hit**, returned straight from the cache. `cache_info()` proves it: 36 real
computations (misses) and 33 instant lookups (hits), so the recursion never blows up.

📝 **`lru_cache`** - "Least Recently Used cache." It remembers a function's results keyed by its
arguments; `maxsize` caps how many it keeps (evicting the least-recently-used), and `maxsize=None` means
unbounded. Arguments must be **hashable** (dicts and lists can't be cache keys), and the function must be
pure - caching one with side effects or time-dependent output hands you stale answers.

**4. Push hot numeric loops into NumPy / C.** When the hot spot is genuinely heavy number-crunching over a
big array - and you've confirmed it with a profile - move it out of pure Python entirely. **NumPy** stores
numbers in a tight, typed C array (no per-number Python object, none of that allocation-and-refcount cost
from the memory section) and runs operations over the whole array in compiled C. A loop that adds a
million numbers one Python object at a time becomes a single vectorized operation. This is the real answer
to "Python is slow for math": you don't speed up Python, you let it *orchestrate* fast C. The same idea
powers pandas, scikit-learn, and the rest of the scientific stack.

**5. Only now, micro-optimize.** Last, and only inside a hot spot a profiler flagged: avoiding repeated
attribute lookups, hoisting work out of a loop, choosing a comprehension. These give small wins in tight,
proven-hot code, and waste your attention and your code's readability everywhere else. Micro-optimization
is the *bottom* of this list for a reason.

> 🪖 **War story.** A teammate was sure a report was slow because of "all those list comprehensions" and
> spent a day rewriting them into loops "for speed." No change. A five-minute `cProfile` run afterward
> showed the whole time was one function calling the database once *per row* - an N+1 query. One fix
> there, batching the query, and the report went from minutes to seconds. The day of comprehension
> rewrites bought nothing; the profiler would have pointed at the real culprit in five minutes.

## Recap

1. Python is "slow" because it's **interpreted and dynamically typed** - types are looked up at runtime
   on every operation. That cost only bites in tight per-operation loops, and most programs are waiting
   on I/O anyway.
2. CPython runs your code in two steps: **source → bytecode → an eval loop** that executes each
   instruction. `dis` lets you see the bytecode; `__pycache__` caches it.
3. Memory is freed by **reference counting** (freed the instant the count hits zero), with a **cyclic
   garbage collector** as backup for reference cycles. The deep version lives in
   [Memory & Garbage Collection](/guides/memory-and-garbage-collection).
4. The cardinal rule: **measure, don't guess.** Use `timeit` for small comparisons and `cProfile` to find
   the real hot spot. Treat any timing number - including ones in guides - as illustrative; run it
   yourself.
5. **Premature optimization** is the trap: rewriting the wrong 5% from a hunch. Profile first, fix only
   the proven hot spot, measure again.
6. The real speedups, in order: **better algorithm/data structure → built-ins & stdlib (C) → cache with
   `lru_cache` → push numeric work into NumPy/C → only then micro-optimize.**

You now know *why* Python runs the way it does and how to make it faster without flailing. Next, the other
side of shipping real Python: getting it, and the exact libraries it depends on, to run reliably on a
machine that isn't yours.

Feel how each complexity class grows:

```playground-bigo
```

Watch reference counting and the cycle collector at work:

```playground-gc
```

Quick check - make sure these stuck:

```quiz
[
  {
    "q": "Why is Python often called \"slow\" compared to a language like C?",
    "choices": [
      "Its syntax is too verbose, so programs take longer to write",
      "It's interpreted and dynamically typed, so types are looked up and dispatched at runtime on every operation",
      "It can only use a single CPU core no matter what",
      "It stores all numbers as text and re-parses them constantly"
    ],
    "answer": 1,
    "explain": "The cost is structural: because Python is interpreted and types aren't known ahead of time, every operation pays for a runtime type lookup and dispatch that a compiled, statically-typed language did once, up front."
  },
  {
    "q": "You think you know which function in your program is the bottleneck. What should you do before rewriting it for speed?",
    "choices": [
      "Trust the hunch and rewrite it - you wrote the code, so you know it best",
      "Rewrite every loop as a comprehension first, since comprehensions are fast",
      "Profile it (e.g. with cProfile) to find where the time actually goes, then optimize only the proven hot spot",
      "Switch the whole program to NumPy to be safe"
    ],
    "answer": 2,
    "explain": "Measure, don't guess. The hot spot is almost never where intuition points, so profile first, fix only the part that's actually slow, and measure again to confirm it helped. Rewriting from a hunch is premature optimization."
  },
  {
    "q": "A profiler has confirmed a genuine hot spot. Which speedup should you reach for first?",
    "choices": [
      "Micro-optimizations like avoiding repeated attribute lookups",
      "A better algorithm or data structure (e.g. a set instead of a list for membership checks)",
      "Rewriting the whole thing in NumPy",
      "Adding @lru_cache to every function"
    ],
    "answer": 1,
    "explain": "The order that pays off: better algorithm/data structure → built-ins & stdlib in C → cache repeated work with lru_cache → push numeric loops into NumPy/C → and only then micro-optimize. A better algorithm dwarfs everything else; no micro-tuning rescues an O(n²) approach when an O(n) one exists."
  }
]
```


---

# Packaging & Environments

You have a folder full of `.py` files that does something useful. A friend asks for it, and your real
answer is "clone this, make a venv, install these three things, and run `python main.py` from the right
directory." That works, barely, and only because you're standing next to them. What you actually want is
for them to type one line - `pip install your-thing` - and have it work on a machine you'll never see.

This phase is about that gap: turning a folder of scripts into a **package** the rest of the world can
install. The mental model first - packaging has a reputation for being a confusing thicket of tools, and
most of that confusion comes from not seeing what each tool is *for*.

## The mental model: source folder → built artifact → index → install

**What it actually is.** Packaging is a small assembly line. Your **source folder** (code plus a
description of the project) gets turned into a **built artifact** - a single file in a standard format -
which gets uploaded to an **index** (a public server). Anyone's `pip` can then download it from the index
and install it. Four stops, one direction.

```mermaid
flowchart LR
  src(your source folder) --> build[python -m build]
  build --> wheel(wheel / sdist)
  wheel --> pypi[(PyPI)]
  pypi --> install[pip install]
```

*One idea:* every tool in this phase lives at exactly one of those arrows. `build` makes the artifact;
`twine` (or `uv`) uploads it to PyPI; `pip` downloads and installs it. Know which arrow a command stands
on, and the landscape stops being a thicket.

> 💡 **Key point.** "Packaging" is two separate jobs people lump together: *isolating* the dependencies a
> project needs (virtual environments), and *distributing* your project so others can install it (build +
> publish). The first you do for every project; the second only when you have something to share.

## Recap from Phase 8: why the virtual environment comes first

You met **virtual environments** in [Phase 8](08-ecosystem-and-tooling.md), and they matter here too - the
one-line refresher: a venv is a private box holding its own Python and packages, isolated from every other
project. You make one per project so project A's `requests` 2.20 can't collide with project B's `requests`
2.32.

📝 **Virtual environment (venv)** - a per-project folder (usually `.venv`) with its own copy of Python and
its own installed packages. Activate it and `pip install` drops things *into that box only*.

The reason it leads this phase: you build and test a package *inside* a clean venv. Build against your
messy global Python and you can't tell which dependencies you actually declared versus which just happened
to be lying around - and your users have a different pile lying around. A fresh venv is the real test of
"did I declare everything this project needs?"

```console
$ python -m venv .venv
$ source .venv/bin/activate          # macOS/Linux  (Windows: .venv\Scripts\activate)
(.venv) $
```
*What just happened:* you created an empty, isolated Python in `.venv` and switched your shell into it -
the `(.venv)` prefix proves you're inside the box. Everything you install from here on lands in this
folder and nowhere else.

## The dependency-tool landscape, plainly

There are two ways people manage a project's dependencies today, and the right answer depends on how much
the project will grow. Here's the plain comparison, both sides, not a sales pitch for either.

| | **`pip` + `venv`** (built-in) | **`poetry`** / **`uv`** (all-in-one) |
|---|---|---|
| **What it is** | The baseline that ships with Python. `venv` makes the box, `pip` installs into it. | A single tool that resolves, installs, *and* manages the venv for you. `uv` is the fast newer one; `poetry` is the established one. |
| **Lockfile** | None built in - you pin by hand with `pip freeze`. | Yes - a lockfile records the *exact* resolved versions of every dependency and sub-dependency. |
| **Resolver** | Installs what you ask, one at a time. | Solves all your dependencies together so they're mutually compatible before installing anything. |
| **You install** | Nothing - it's already there. | An extra tool, once. |
| **Good when** | Small project, scripts, learning, or you want zero extra tooling. | Real project with many deps, a team, or reproducible builds that matter. |

📝 **Lockfile** - a file recording the *exact* version of every package installed, including
dependencies-of-dependencies. `requirements.txt` from `pip freeze` is a hand-rolled version of this; tools
like `poetry` and `uv` generate and update one automatically.

📝 **Resolver** - the part that figures out a set of versions that all work together. If package A needs
`urllib3<2` and package B needs `urllib3>=2`, a real resolver tells you *before* installing; naive
installation discovers it only when something breaks at runtime.

**The straight take.** For your first shareable package, plain `pip` + `venv` + a `pyproject.toml` is
completely enough, and it's the foundation everything else is built on - so that's what we'll use below.
If you later fight dependency conflicts or want one-command reproducible installs, reach for `uv` or
`poetry`; they automate the same `pyproject.toml` you're about to write, plus the lockfile. Nothing here
is wasted by moving to them.

## `pyproject.toml` - the modern project descriptor

**What it actually is.** `pyproject.toml` is the one file that describes your project: its name, version,
what it depends on, and how to build it. It's a Python standard (every modern tool reads the same file),
and replaces the older scattered setup (`setup.py`, `setup.cfg`, and friends) with a single declarative
document.

📝 **TOML** - a plain, readable config format: `key = value`, grouped under `[section]` headers. No code,
no surprises - it's data, not a script.

Here is a complete, real one for a small command-line tool:

```toml
[project]
name = "greet-cli"
version = "0.1.0"
description = "A tiny CLI that greets people."
readme = "README.md"
requires-python = ">=3.10"
dependencies = [
    "requests>=2.32",
]

[project.scripts]
greet = "greet_cli.main:run"

[build-system]
requires = ["hatchling"]
build-backend = "hatchling.build"
```

*What just happened:* this file says four things. The `[project]` table is the identity card - name,
version, one-line description, minimum Python, runtime dependencies. `[project.scripts]` wires up a
terminal command: after install, typing `greet` runs the `run` function inside `greet_cli/main.py`.
`[build-system]` names the **build backend** - the tool that turns your source into an artifact (here,
`hatchling`, a common low-fuss choice; `setuptools` is the classic alternative).

📝 **Build backend** - the engine that reads your `pyproject.toml` and produces the wheel/sdist. You rarely
interact with it directly; the front-end tool (`python -m build`) calls it for you. You only name which one
in `[build-system]`.

⚠️ **Gotcha - `name` vs. import name.** The `name` on PyPI (`greet-cli`, with a hyphen) and the name you
`import` in code (`greet_cli`, with an underscore - Python identifiers can't contain hyphens) are *not*
required to match, and beginners conflate them constantly. Pick the distribution `name` for the index and
make sure your package folder uses a valid import name. Keeping them parallel (`greet-cli` ↔ `greet_cli`)
saves everyone the headache.

## Building a distributable - `python -m build`

**What it actually is.** Building takes your source folder and produces the artifact that gets shipped.
There are two artifact types, and you almost always make both:

📝 **Wheel** (`.whl`) - the *built*, ready-to-install format. `pip` prefers it because it's pre-assembled:
no build step on the user's machine, it just unpacks. This is what most people install.

📝 **sdist** (source distribution, `.tar.gz`) - your source code in a tarball. The fallback when no
compatible wheel exists, and good provenance to publish alongside the wheel.

The standard front-end tool is `build`. Install it into your venv, then run it from your project root:

```console
(.venv) $ python -m pip install build
(.venv) $ python -m build
* Creating isolated environment...
* Building sdist...
* Building wheel...
Successfully built greet_cli-0.1.0.tar.gz and greet_cli-0.1.0-py3-none-any.whl
```
*What just happened:* `build` read your `pyproject.toml`, called the build backend in a clean isolated
environment, and dropped two files into a new `dist/` folder - the sdist (`.tar.gz`) and the wheel
(`.whl`). `py3-none-any` in the wheel name means "pure Python, any platform, any CPU" - installs anywhere
Python 3 runs. Those two files in `dist/` are the entire thing you ship.

## Publishing to PyPI - so others can `pip install`

**What it actually is.** **PyPI** (the Python Package Index) is the public server `pip` downloads from.
Publishing means uploading your `dist/` files there - once they land, your package is a name anyone in
the world can `pip install`.

📝 **PyPI / TestPyPI** - PyPI is the real, public index. **TestPyPI** is a separate sandbox copy for
rehearsing a publish without burning a version number on the real thing. Practice on TestPyPI first.

The classic uploader is **twine**:

```console
(.venv) $ python -m pip install twine
(.venv) $ python -m twine upload dist/*
Uploading distributions to https://upload.pypi.org/legacy/
Uploading greet_cli-0.1.0-py3-none-any.whl
100% ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 8.2/8.2 kB
Uploading greet_cli-0.1.0.tar.gz
100% ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 6.1/6.1 kB

View at:
https://pypi.org/project/greet-cli/0.1.0/
```
*What just happened:* `twine` uploaded both files from `dist/` to PyPI and printed the live URL for your
new release. It'll prompt for an API token the first time (generate one in your PyPI account settings -
username/password uploads are no longer accepted). From this moment, `pip install greet-cli` works for
everyone.

If you're using `uv`, it's the same two steps under one tool - `uv build` writes the wheel and sdist to
`dist/`, then `uv publish` uploads them. Same destinations, same result; `uv build` replaces `python -m
build` and `uv publish` replaces `twine upload`.

> 🪖 **War story - the version you can't take back.** PyPI won't let you re-upload a version number once
> it's published; `0.1.0` is `0.1.0` forever, even if you spot a typo thirty seconds later. The fix is
> always *forward* - bump to `0.1.1` and publish again. That's exactly what TestPyPI is for: make your
> embarrassing mistakes in the sandbox, where burning a version number costs nothing.

## The editable install - `pip install -e .`

**The problem it solves.** While you're *developing* the package, you don't want to rebuild and reinstall
after every code change - you want your installed package to *be* your source folder, so edits show up
instantly.

**What it actually is.** An **editable install** installs a link to your source directory instead of a
copy of it. `import greet_cli` then loads your live files - edit a function, rerun, and the change is
already there.

```console
(.venv) $ python -m pip install -e .
Obtaining file:///home/you/greet-cli
Installing build dependencies ... done
Successfully installed greet-cli-0.1.0
```
*What just happened:* the `-e` (editable) flag plus `.` (this directory) installed your project as a link
back to the source folder, not a frozen copy. Now `import greet_cli` and the `greet` command both run your
current code - the install you use *while building*; the wheel is what users get when you're done.

⚠️ **Gotcha - keep build junk and secrets out of git.** Building creates `dist/`, often `build/`, and a
`*.egg-info/` folder; your venv lives in `.venv/`. None of that belongs in version control - it's
generated, machine-specific, and bloats the repo. Add a `.gitignore`:

```console
$ cat .gitignore
.venv/
dist/
build/
*.egg-info/
__pycache__/
```
*What just happened:* git now ignores the generated artifacts and your local environment, so a clean clone
contains only source. The security half of the same rule: your PyPI API token, and any credentials a tool
stores in its config, are **secrets** - never commit them. Keep tokens in an environment variable or your
tool's credential store, never pasted into a tracked file.

## Recap

1. Packaging is an assembly line with one direction: **source folder → built artifact → PyPI → `pip
   install`**. Every command lives on exactly one of those arrows.
2. A **virtual environment** (from [Phase 8](08-ecosystem-and-tooling.md)) isolates each project's deps -
   build and test inside a clean one so you know you've declared everything.
3. The tool landscape, plainly: **`pip` + `venv`** is the built-in baseline and enough for a first
   package; **`poetry`/`uv`** add lockfiles and one-command resolve-install-venv when a project grows.
4. **`pyproject.toml`** is the modern, standard descriptor - name, version, dependencies, and build-system
   in one declarative file.
5. **`python -m build`** produces a **wheel** (built, ready-to-install) and an **sdist** (source tarball)
   into `dist/`.
6. Publish with **`twine upload`** (or **`uv publish`**) to **PyPI** - and versions are permanent, so
   rehearse on **TestPyPI** first.
7. **`pip install -e .`** gives you an editable install for development; keep `dist/`, `build/`,
   `*.egg-info/`, and `.venv/` out of git, and never commit secrets.

You can now hand someone a package name instead of a list of instructions. Next, the guide steps back from
your own code to the wider world - the libraries, communities, and directions worth knowing as you keep
going with Python.

Quick check - see if the assembly line and the isolation rule stuck:

```quiz
[
  {
    "q": "Why make a fresh virtual environment per project before building a package?",
    "choices": [
      "It makes pip install run faster",
      "It isolates each project's dependencies so you can tell exactly what you declared versus what was just lying around globally",
      "PyPI refuses uploads built outside a venv",
      "Virtual environments are required to write a pyproject.toml"
    ],
    "answer": 1,
    "explain": "A venv is a private box with its own Python and packages. Building inside a clean one is the real test of whether you declared every dependency, instead of relying on whatever happens to be installed globally."
  },
  {
    "q": "Which file is the modern, standard descriptor for a Python project - its name, version, dependencies, and build backend?",
    "choices": [
      "requirements.txt",
      "setup.py",
      "pyproject.toml",
      ".gitignore"
    ],
    "answer": 2,
    "explain": "pyproject.toml is the single declarative file every modern tool reads. It replaces the older scattered setup.py / setup.cfg and names the build backend in [build-system]."
  },
  {
    "q": "What's the right way to get a package into other people's hands, and what stays out of git?",
    "choices": [
      "Email them your .py files; commit .venv/ and dist/ so they have everything",
      "python -m build makes a wheel, twine (or uv) publishes it to PyPI, and others pip install it - while .venv/, dist/, build/, and secrets stay out of git",
      "Push your repo to GitHub; pip install reads straight from the default branch",
      "Run pip install -e . on their machine over SSH"
    ],
    "answer": 1,
    "explain": "The assembly line is source → build (wheel/sdist) → publish to PyPI → others pip install. Generated artifacts (.venv/, dist/, build/, *.egg-info/) and API tokens are machine-specific or secret, so they belong in .gitignore, never the repo."
  }
]
```


---

# Where to Go Next

You've gone the whole distance - from "what is Python" through classes and errors, into the deep half:
the data model, generators, decorators, typing, concurrency, performance, packaging. That's the *language*
genuinely in your hands. What's left isn't *more Python* so much as *Python pointed at a problem*, and
which direction you go depends entirely on what you want to make.

So this last phase is a map, not a curriculum - no "you must learn all of these." Pick the branch that
matches what you're building, ignore the rest until you need them, and, most importantly, go build the
thing.

## The four big branches

Each of these is a whole world. Here's the plain one-paragraph version of each, so you can tell which
one is yours.

```mermaid
flowchart TD
  P(You know Python) --> W[Web]
  P --> D[Data]
  P --> A[Automation]
  P --> K[Packaging]
  W --> WT(Django / FastAPI)
  D --> DT(pandas / NumPy)
```

*One idea:* the same Python you learned branches toward different goals - a website, a data analysis, a
script that does your chores, a tool you share. Same language, different destination.

### Web - make something people open in a browser

If you want to build a website or an API other programs call, this is your branch. The two names you'll
hear most:

- **Django** - a "batteries-included" framework. It hands you an admin panel, user accounts, database
  models, and forms out of the box - great for a full application where you want the common parts already
  solved. The trade-off: opinionated and large, so there's more to learn up front.
- **FastAPI** - a modern, lightweight framework focused on building APIs (the JSON-over-HTTP kind). Smaller,
  fast to start with, and leans on the type hints from [Phase 14](14-type-hints.md). Great when you want an
  API and not a whole website.

Neither is "better" - Django is more *included*, FastAPI is more *minimal*. To understand what an API even
is before you pick, [What an API Is](/guides/what-an-api-is) is the grounding. (Deep dives on both are
their own guides.)

### Data - turn numbers into answers

If your goal is analysis - spreadsheets too big for Excel, charts, models - this is where Python genuinely
dominates.

- **NumPy** - fast numerical arrays. The foundation almost everything data-related is built on (and the
  practical escape hatch from the performance limits of [Phase 17](17-performance-and-memory.md)).
- **pandas** - tables (it calls them DataFrames) with filtering, grouping, and joining, built on NumPy. If
  you've ever wished a spreadsheet were programmable, this is that.

This branch is deep (it leads toward machine learning), but pandas alone will already change how you
handle any pile of data.

### Automation - make the computer do your chores

The least glamorous branch and often the most immediately useful: small scripts that rename files, scrape
a page, send a report, or poke an API on a schedule. You can start *today* with only what this guide
taught you plus the `requests` library from [Phase 8](08-ecosystem-and-tooling.md) - this is where most
people feel Python "click," because the payoff is a real chore that never bothers you again.

### Packaging - share what you built

You've already seen the mechanics in [Phase 18](18-packaging-and-environments.md) - `pyproject.toml`,
building, publishing. The "next step" isn't learning *how*, it's having something worth sharing. Don't
rush it - package a tool once it's genuinely useful to someone other than you.

## The no-nonsense advice: build, then look things up

> 💡 **Key point.** You don't learn the next layer by reading about it. You learn it by trying to build
> something that needs it, getting stuck, and looking up exactly the piece you're stuck on. A tutorial you
> follow start-to-finish teaches you to follow tutorials; a project you fight through teaches you to build.

A few starter projects sized to where you are right now:

- **Automation:** a script that reads a folder of files and renames them by a rule you choose.
- **Web:** a FastAPI app with one endpoint that returns some JSON. Just one. Then add a second.
- **Data:** load a CSV with pandas, filter it, and print the answer to one question you actually care
  about.

Pick the smallest version of the thing you want to exist and build *that*. Everything in this guide -
types, collections, functions, classes, the data model, generators, decorators, typing, concurrency - was
the vocabulary. A project is where it becomes fluency.

## One last reframe

Python was a deliberate choice as a first language: readable, forgiving, useful in nearly every corner of
software. But the *ideas* you picked up here - variables and types, collections, control flow, objects,
the data model, iteration, error handling, concurrency - aren't Python's. They're how nearly every modern
language works, dressed in different syntax. Pick up a second language and you'll find you already know
most of it; you're just learning new spellings.
[Languages, Explained Like a Human](/guides/languages-explained-like-a-human) is the map of that bigger
landscape, for whenever you're curious what else is out there.

You came in not knowing what `print("hello")` did. You're leaving able to reason about an entire program
*and* the runtime underneath it. Go make something.

## Recap

1. Python branches toward **web** (Django for full apps, FastAPI for APIs), **data** (NumPy + pandas),
   **automation** (small useful scripts), and **packaging** (sharing what you built).
2. Pick the *one* branch matching what you want to make; ignore the rest until you need them.
3. You learn the next layer by **building something and looking up what you get stuck on**, not by
   reading ahead.
4. The concepts you learned are nearly universal across languages; a second language is mostly new
   spelling.

One last check - the through-lines of the whole guide:

```quiz
[
  {
    "q": "Python branches toward different goals. If you wanted to build an API that other programs call over HTTP, which branch is that?",
    "choices": ["Data (NumPy / pandas)", "Web (Django / FastAPI)", "Packaging (pyproject.toml)", "Automation (small scripts)"],
    "answer": 1,
    "explain": "Web is the branch for websites and APIs - Django for full applications, FastAPI for lightweight APIs. Data, automation, and packaging point at different destinations."
  },
  {
    "q": "What's the guide's no-nonsense advice for learning the next layer beyond this guide?",
    "choices": ["Read every framework's docs cover to cover first", "Follow a long tutorial start to finish before building anything", "Build something that needs it, get stuck, and look up exactly the piece you're stuck on", "Memorize the standard library before starting a project"],
    "answer": 2,
    "explain": "You learn by building and looking things up when you get stuck. A tutorial teaches you to follow tutorials; a project you fight through teaches you to build."
  },
  {
    "q": "You finish this guide and later pick up a second language. What carries over?",
    "choices": ["Almost nothing - every language is entirely its own world", "The core ideas - variables, collections, control flow, objects, iteration, errors - which are nearly universal", "Only Python's exact syntax, which you'll have to unlearn", "Just the print statement"],
    "answer": 1,
    "explain": "The concepts you learned aren't Python's - they're how nearly every modern language works, dressed in different syntax. A second language is mostly learning new spellings."
  }
]
```