Celery From Zero
Some work is too slow to do during a web request: sending an email, generating a report, processing an upload, calling a slow third-party API. Make the user wait for it and your app feels broken. Celery is the answer the Python world reaches for — a distributed task queue that lets your app hand a job off to run later, on a separate pool of worker processes, while the request returns instantly. FastAPI, Django, and Flask apps all lean on it for exactly this.
The mental model that makes Celery click is a four-part hand-off: your app puts a task message on a broker (a queue, usually Redis or RabbitMQ); one of several workers picks it up and runs it; and an optional result backend stores what it returned. Once you see those four pieces — and that they're separate processes talking through a queue — Celery stops being intimidating configuration and becomes a shape you can reason about (and debug).
📝 This assumes Python (Python From Zero) and pairs naturally with the queue concepts in Webhooks & Message Queues. It's the background worker that the framework guides (FastAPI, Django, Flask) all hand heavy jobs to. Celery needs a broker + worker processes, so examples are shown with the commands to run them yourself.
How to read this
Read in order — it builds from a single background task up to scheduled jobs and a monitored production setup, using a running example of a web app offloading email and report work. Phases carry difficulty badges.
The phases
Part 1 — The core model (🟢 Basic → 🟡)
- What Celery Is & Why 🟢 — the problem, and the broker/worker/result mental model.
- The Broker & Worker 🟢 — the queue (Redis/RabbitMQ) and the worker processes that drain it.
- Defining & Calling Tasks 🟡 —
@task,.delay()/.apply_async(), and how a job gets queued and run.
Part 2 — Doing it right (🟡 → 🔴)
4. Results & State 🟡 — the result backend, AsyncResult, and tracking task status.
5. Retries & Error Handling 🔴 — retrying failures, idempotency, and not losing work.
6. Scheduled Tasks with Celery Beat 🟡 — periodic, cron-like jobs.
Part 3 — Production (🔴 → 🟢) 7. Production: Scaling, Monitoring & Pitfalls 🔴 — concurrency, Flower, and the mistakes that bite everyone. 8. Where to Go Next 🟢 — Celery vs the alternatives, framework integration, and what to build.
The whole thing is a hand-off: app → broker → worker → (result). Hold those four pieces and Celery's configuration and quirks all fall into place.