Operating System Essentials for System Design

TL;DR: Your database is slow because of disk seeks. Your API is slow because of blocking I/O. Your service falls over at 10K connections because you used threads instead of an event loop. The OS is the floor every system design sits on, and the gaps between its layers span eight orders of magnitude.

Learning Objectives

After this module, you will be able to:

• Explain the cost difference between a process and a thread context switch
• Recite the memory hierarchy with order-of-magnitude latency numbers
• Pick the right I/O model for a workload (blocking, multiplexed, or async)
• Describe what the page cache does and when it lies to you about durability
• Reason about why fsync is expensive and how group commit fixes it
• Identify when kernel bypass is justified vs overkill

Intuition

Think of your computer as a factory with a loading dock. The CPU is the assembly line: blindingly fast, but it can only work on materials already on the bench. Main memory is the warehouse next door: 100x slower to fetch from than the bench, but it holds a lot. The SSD is a storage unit across town: 1,000x slower still, but massive. And a cross-continent network call is like ordering parts from another country: 300,000x slower than grabbing something off the bench.

Every performance problem in distributed systems traces back to someone accidentally sending the assembly line to the storage unit across town when the part was already in the warehouse. Or worse, sending it overseas for every single bolt.

This chapter gives you the mental model: what a thread actually costs, why epoll exists, what the page cache is doing behind your read() call, and how production systems like Redis and Kafka exploit these layers to move millions of operations per second on commodity hardware.

Theory

Processes vs threads

A process is an isolated execution environment: its own virtual address space, page tables, file descriptor table, and memory protections. Two processes cannot read each other’s memory without explicit IPC.

A thread is a flow of execution inside a process. Threads share heap, globals, and file descriptors. On Linux, both are created via clone(2) with different flags: threads share CLONE_VM and CLONE_FILES, which makes creation and switching cheaper because the TLB (translation lookaside buffer) stays warm ¹.

A context switch saves one task’s registers and restores another’s. Between threads in the same process this costs roughly 1 microsecond. Between processes it costs 3 to 5 microseconds because the TLB must flush ².

Memory hierarchy

These are the numbers you should know cold. They are based on Jeff Dean’s canonical “Latency Numbers Every Programmer Should Know” ³, updated for modern NVMe storage ⁴:

flowchart LR
    L1["L1 Cache\n0.5 ns\n64 KB"] --> L2["L2 Cache\n7 ns\n256 KB"]
    L2 --> RAM["Main Memory\n100 ns\n32-256 GB"]
    RAM --> SSD["NVMe SSD\n20-70 us\n1-8 TB"]
    SSD --> HDD["HDD Seek\n10 ms\n10+ TB"]
    RAM --> DC["Same DC RTT\n500 us"]
    DC --> WAN["Cross-continent\n150 ms"]

The practical lesson: keep hot data in RAM. If you cannot, keep it on NVMe. If you cannot, design around the 10 ms cost of a disk seek with batching, pipelining, and async I/O.

I/O models and the C10K problem

In 1999, Dan Kegel asked: how does one machine serve 10,000 concurrent connections ⁵? The answer drove the adoption of event-driven I/O and shaped every modern server.

Blocking I/O (thread-per-connection). Your thread parks in the kernel until data arrives. Simple code, easy debugging. Falls over past a few thousand connections because each thread reserves ~8 MB of stack (Linux default ulimit -s, mostly virtual) and adds context-switch overhead.

I/O multiplexing (event loop). One thread watches many file descriptors. The kernel tells you which are ready, and you handle them in a loop. select is O(n) and capped at 1,024 fds. poll removes the cap but stays O(n). epoll (Linux 2.5.44+, October 2002) keeps the watch set in kernel state and returns only ready fds in O(ready) time ⁵. kqueue (FreeBSD, macOS) provides the same pattern on BSD systems.

Async I/O (io_uring). Linux io_uring (kernel 5.1, 2019) uses two shared ring buffers: a submission queue and a completion queue. Userspace and kernel pass operations without a syscall per op, achieving multi-x throughput over syscall-per-op patterns in microbenchmarks ⁶⁷.

flowchart TD
    A["New service:\nwhat I/O model?"] --> B{"Concurrent\nconnections?"}
    B -->|"< 1K"| C["Thread-per-connection\nSimpler code, fine here"]
    B -->|"1K to 100K+"| D{"Platform?"}
    D -->|Linux| E["epoll event loop\nor async runtime"]
    D -->|"BSD / macOS"| F["kqueue event loop"]
    B -->|"High-throughput\ndisk + network"| G{"Linux 5.1+?"}
    G -->|Yes| H["io_uring\n(trusted hosts only)"]
    G -->|No| E
    B -->|"C10M:\nline-rate 100GbE+"| I["Kernel bypass\nDPDK / XDP"]

File systems and the page cache

A file system organizes bytes on a block device into files and directories. An inode holds a file’s metadata and pointers to data blocks. Directories map names to inode numbers.

Journaling protects against corruption after a crash. Before modifying metadata, the FS writes a journal entry describing the intent. After a crash, recovery replays the journal. ext4 defaults to data=ordered mode: metadata is journaled, data is written before the metadata commit ⁹.

The page cache is the kernel’s RAM cache of file contents. Reads hit it when warm (nanoseconds). Writes land in dirty pages that flush to disk later. This is why your second benchmark run is 10 to 100x faster than the first: the first run populated the cache from disk.

sequenceDiagram
    participant App
    participant Kernel as Kernel (VFS)
    participant Cache as Page Cache (RAM)
    participant Disk as NVMe / HDD
    App->>Kernel: read(fd, buf, 4096)
    Kernel->>Cache: lookup page
    alt Cache hit (warm)
        Cache-->>Kernel: bytes (~100 ns)
        Kernel-->>App: 4096 bytes
    else Cache miss (cold)
        Cache-->>Kernel: not present
        Kernel->>Disk: block read
        Disk-->>Kernel: bytes (NVMe ~50 us, HDD ~10 ms)
        Kernel->>Cache: populate page
        Kernel-->>App: 4096 bytes
    end

Durability requires fsync. A write() returns when bytes reach the page cache, not the disk. Durability demands fsync(), which forces the device to flush its write cache to stable media: 1 to 10 ms on NVMe, 10 to 30 ms on HDD ⁴. One fsync per write caps you at a few hundred durable writes per second. The fix is group commit: buffer writes, append them as a batch, call fsync once, then acknowledge every client in the batch.

Syscalls and kernel cost

A syscall is a privilege transition: save user registers, enter kernel mode, do work, return. The round trip on x86_64 Linux is hundreds of nanoseconds ¹⁰. Hot-path calls like clock_gettime use the vDSO (virtual dynamic shared object), a kernel-mapped page that lets userspace read kernel data without entering kernel mode, dropping the cost to 20 to 30 ns ¹⁰.

Any code that assumes syscalls are free hits the wall fast. A per-message write() to a socket caps at a few hundred thousand calls per second. Batching (sendmmsg, io_uring submission rings) amortizes the transition cost across many operations ⁶.

Real-World Example

Redis: a single-threaded event loop serving 100K ops/sec.

Redis wraps all I/O multiplexing behind a small abstraction called ae (Async Event). At compile time, it selects the best available backend: evport (Solaris), then epoll (Linux), then kqueue (BSD/macOS), then select as the fallback ¹¹.

The main loop calls epoll_wait (on Linux), which blocks until at least one fd is ready or a timer fires. For each ready fd, Redis invokes the registered read or write callback. The entire command execution is single-threaded: no locks on data structures, no race conditions, no context switches between commands ¹².

This works because each Redis command is O(1) or O(log n) and completes in microseconds. The bottleneck is network or memory bandwidth, not CPU. A single Redis thread handles ~100K operations per second and tens of thousands of concurrent connections on commodity hardware ¹².

sequenceDiagram
    participant C1 as Client 1
    participant C2 as Client 2
    participant Redis as Redis (single thread)
    participant Kernel as epoll (kernel)
    Redis->>Kernel: epoll_wait(fds, timeout)
    Kernel-->>Redis: fd 5 readable, fd 9 readable
    C1->>Redis: GET user:42
    Redis-->>C1: "Alice"
    C2->>Redis: SET counter 7
    Redis-->>C2: OK
    Note over Redis: No context switch,<br/>no lock, no thread pool

Netflix’s Open Connect CDN pushes this further: FreeBSD boxes running custom serving software with kqueue and kernel TLS have scaled from 200 Gbps per server (2020) to 400 Gbps (2021) to 800 Gbps per server (2022) of TLS video ¹³¹⁴¹⁵. They treat the OS page cache as their content cache, use sendfile(2) for zero-copy transfers, and tune NUMA topology to reduce cross-domain memory traffic. The progression from 100 Gbps (2017) to 800 Gbps (2022) on a single box demonstrates that every OS layer is tunable when you understand it.

Trade-offs

Common Pitfalls

Exercise

Design Challenge: You are writing a log ingestion service that needs to accept 100K writes per second from clients and durably persist each write before acknowledging. Each write is ~1 KB. What I/O model do you pick? How do you make fsync cheap?

Key Takeaways

• Threads share memory and switch in ~1 us; processes are isolated and switch in ~3-5 us. Use the lightest abstraction that gives you the isolation you need.
• The memory hierarchy spans 8 orders of magnitude: L1 at 0.5 ns to cross-continent at 150 ms. Design around the gaps.
• For 1K+ concurrent connections, use an event loop (epoll/kqueue). Thread-per-connection does not scale past that.
• The page cache makes reads fast but lies about durability. Only fsync guarantees bytes are on disk, and it costs 1 to 10 ms.
• Group commit turns expensive per-write fsyncs into cheap per-batch fsyncs, unlocking 100x throughput.
• io_uring is the future of Linux async I/O but carries security risk. Use it on trusted hosts for high-throughput workloads.
• Redis proves that a single-threaded event loop can handle 100K ops/sec when each operation is fast and the I/O model is right.

Further Reading

• Latency Numbers Every Programmer Should Know - Jeff Dean’s canonical table; memorize the order-of-magnitude gaps between tiers before any capacity estimation.
• The C10K Problem - Dan Kegel’s 1999 manifesto that drove the adoption of epoll and kqueue; still the best framing of why thread-per-connection fails.
• Efficient IO with io_uring - Jens Axboe’s design paper explaining submission/completion rings; read this before adopting io_uring in production.
• Serving 100 Gbps from an Open Connect Appliance - Netflix’s page-cache-centric CDN design showing how OS tuning unlocks extreme throughput.
• Apache Kafka Design - the “Don’t fear the filesystem” section explains why sequential writes, page cache, and sendfile make Kafka fast.
• Redis event library: ae.c - a small, readable cross-platform event loop; study ae_epoll.c to see how thin the epoll wrapper really is.
• Operating Systems: Three Easy Pieces - free textbook covering processes, virtual memory, and concurrency; the right follow-up if this chapter moves too fast.
• Socket Sharding in NGINX Release 1.9.1 - measured 2-3x throughput and 30% lower latency with SO_REUSEPORT; a concrete example of OS-level tuning.

Flashcards

References