Build Your Own Key-Value Store (Python)

Build a tiny persistent key-value database from scratch in Python - an append-only log, crash recovery, an in-memory index, compaction, and a TCP server - and come out understanding how real storage engines actually work.

1. A Dict, and Why Persistence Is Hard The in-memory store is ten lines of Python; saving it to disk naively fails two ways - every write rewrites everything, and a crash mid-write corrupts the whole file.
2. The Append-Only Log - the Write Path Design a binary record format with length prefixes and a CRC32 checksum, append every change to a log, and learn what flush and fsync actually promise about your data reaching the disk.
3. Replay and the Index - the Read Path Rebuild the store's state by replaying the log on startup, recover cleanly from a record that was half-written during a crash, and swap values-in-RAM for a real index of file offsets.
4. Compaction - the Log Can't Grow Forever An append-only log accumulates every dead version of every key; compaction rewrites only the live records to a new file and swaps it in atomically, reclaiming the space without losing a byte.
5. A TCP Server - Speaking to Other Programs Put a SET/GET/DEL wire protocol in front of the engine with a threaded TCP server and a lock, so any program on the machine can use your database - and see why concurrent writers need serializing.
6. Benchmarks, and What Redis Does Differently Measure your engine's real throughput, see the brutal price of fsync, and place your design honestly next to Redis, LevelDB, and SQLite - what each one changed and what it cost them.