Scalability: Growing a System Without Breaking It

TL;DR: Scalability is the ability to handle growth without a rewrite. You have two levers: vertical (bigger machine) and horizontal (more machines). Vertical is simpler but hits a hard ceiling at 896 vCPUs on the largest AWS instance¹. Horizontal removes the ceiling but demands stateless design, because any replica must handle any request². The discipline is knowing which lever to pull next, and knowing when the real fix is a faster query rather than more infrastructure.

Learning Objectives

After this module, you will be able to:

• Explain vertical vs horizontal scaling with concrete cost and ceiling numbers
• Recognize stateful services and know how to make them stateless
• Separate read scaling (replicas, caches, CDNs) from write scaling (sharding, batching)
• Sketch a scaling roadmap from 1 user to 1 billion users
• Identify situations where scaling is the wrong answer and profiling is the right one
• Describe how Shopify, Figma, and Notion each chose different scaling strategies for different reasons

Intuition

Imagine you run a coffee shop. One barista serves 30 customers per hour. Demand doubles. You have two options: hire a faster barista who can pull shots in half the time (vertical scaling), or add a second barista (horizontal scaling).

The faster barista costs more per hour, has a biological ceiling, and when she calls in sick you serve zero customers. Two baristas cost less per unit of work, can absorb a longer queue, and keep serving when one leaves. But two baristas need a system: who takes the next order? How do they share the espresso machine? What happens when they give a customer conflicting answers about the menu?

That coordination system is the price of horizontal scaling. You pay it in complexity, not hardware. Every real system hits this fork. The discipline of scalability is knowing which lever to pull next, and, just as importantly, knowing when the real fix is a faster recipe (a missing index, a bad query, a feature you should delete) rather than more baristas.

Theory

Vertical vs horizontal scaling

Vertical scaling (scale up) means running your workload on a more powerful machine. Your code does not change. Stack Overflow ran their entire Q&A network on 11 web servers (9 primary) backed by SQL Server boxes with 384 GB RAM and 4 TB PCIe SSD, serving 209 million HTTP requests per day and 5.8 billion Redis operations per day³. That is textbook vertical scaling, and it worked for over a decade.

Horizontal scaling (scale out) means distributing the workload across more machines in parallel. Your code must tolerate concurrency, partial failure, and distributed state. But each extra machine has the same price, and there is no ceiling.

Within a general-purpose AWS EC2 family like m7i, the cost per vCPU is roughly flat from the smallest to the largest instance⁴. Vertical scaling does not punish you inside the sweet spot. But the largest high-memory instances (U7i family, 896 vCPUs, 32 TiB RAM) are specialty SKUs for SAP HANA workloads that cost significantly more per vCPU than general-purpose¹⁵. The price curve bends exactly when you most need it.

The ceiling hits in three forms:

1. Physical limits. You cannot buy a 10,000-core box. The largest instance tops out at 896 vCPUs¹.
2. Blast radius. One big machine that dies takes 100% of traffic. Ten small machines that lose one take 10%.
3. Deploy risk. Restarting a vertical giant is scary. Restarting one of a hundred small replicas is routine.

flowchart LR
    subgraph Vertical["Vertical (scale up)"]
        V[Big Server<br/>96 cores, 384 GB RAM<br/>single failure domain]
    end
    subgraph Horizontal["Horizontal (scale out)"]
        H1[App 1] --- H2[App 2] --- H3[App 3]
        H4[App 4] --- H5[App 5] --- H6[App 6]
    end

Stateless vs stateful services

A stateless service holds no per-client information between requests. Any replica can handle any request. The Twelve-Factor App puts it plainly: “Any data that needs to persist must be stored in a stateful backing service, typically a database,” and sticky sessions “are a violation of twelve-factor and should never be used or relied upon”².

A stateful service remembers something about the client or owns a partition of data. A database primary owns the write log. A Kafka broker owns its partitions. A WebSocket gateway owns the TCP socket to a browser.

flowchart LR
    Client --> LB[Load Balancer]
    LB --> A1[App A]
    LB --> A2[App B]
    LB --> A3[App C]
    A1 --> DB[(Redis / Postgres / S3)]
    A2 --> DB
    A3 --> DB

In Kubernetes, this split is encoded in two workload resources. Deployment treats pods as interchangeable (stateless). StatefulSet gives each pod a stable network identity and a persistent volume that follows it across rescheduling, which is what databases, brokers, and distributed KV stores need⁷.

Making a service stateless is usually a matter of moving state out:

• HTTP sessions go into Redis or a signed cookie (JWT).
• In-memory caches become shared caches (Redis, Memcached).
• Background job state moves into a queue (SQS, Kafka) or a database.
• File uploads go to object storage (S3) instead of local disk.

Read scaling vs write scaling

Most systems are read-heavy. Feeds, product catalogs, profile pages: reads outnumber writes by 10x to 10,000x. That asymmetry lets you use fundamentally different tools for each path.

Read scaling is straightforward because reads are idempotent and can be cached or replicated:

• Replicas. A primary streams its write-ahead log to followers. Read throughput scales linearly with replica count⁸.
• Caches. A Redis or Memcached layer absorbs the hot 20% of keys. Stack Overflow’s Redis cluster handled 5.8 billion ops per day at below 2% CPU per instance³.
• CDNs. Push static and edge-cacheable content geographically close to readers. Shopify pushed 12 TB per minute through its edge layer on Black Friday 2024⁹.
• CQRS. Split the write model from one or more read models, each optimized for its query pattern.

Write scaling is hard because writes must land somewhere authoritative:

• Sharding. Split the dataset by a key so different writes land on different nodes. Writes scale linearly with shard count, but cross-shard queries become painful. Notion sharded Postgres into 480 logical shards across 32 physical databases¹⁰. Instagram packed a 13-bit shard ID into every 64-bit primary key so routing requires no external lookup¹¹.
• Batching. Buffer writes in Kafka or a queue and apply them in bulk. Throughput goes up; end-to-end latency goes up too.
• LSM-tree storage. Engines like ScyllaDB and Cassandra absorb writes into a memtable and flush sorted files to disk. Discord migrated from 177 Cassandra nodes to 72 ScyllaDB nodes while dropping p99 read latency from 40-125 ms to 15 ms¹².

sequenceDiagram
    participant Client
    participant Cache as Redis Cache
    participant Replica as Read Replica
    participant Primary as Primary DB

    Client->>Cache: GET user:42
    alt cache hit
        Cache-->>Client: return cached value
    else cache miss
        Cache->>Replica: SELECT * FROM users WHERE id=42
        Replica-->>Cache: row data
        Cache-->>Client: return value (now cached)
    end

    Note over Client,Primary: Write path is separate
    Client->>Primary: UPDATE users SET name='...' WHERE id=42
    Primary-->>Client: ACK
    Primary-)Replica: WAL stream (async)
    Primary-)Cache: invalidate user:42

The scaling roadmap

Real systems do not jump from one server to a globally sharded fleet overnight. The observed pattern across Stack Overflow, Shopify, Figma, and Notion is a staged progression where each move buys 2 to 10x runway:

1. Single box. One app server, one database on the largest available instance. This is where you start.
2. Add a cache. Redis or Memcached in front of the database absorbs repeated reads.
3. Add read replicas. Stream the primary’s WAL to followers; route reads there.
4. Add a CDN. Push static assets and edge-cacheable API responses to the edge.
5. Vertical partitioning. Move table groups onto dedicated databases (Figma went from 1 RDS instance to 12 vertical partitions, extending runway by years before sharding⁶).
6. Horizontal sharding. Split the highest-write tables by a domain-appropriate key. This is the last resort.

Shopify was already on sharded MySQL by 2015 and only introduced pods (isolated datastore slices) in 2016, after realizing that any single shard going down would make that action unavailable across the entire platform¹³. Notion ran a single “beefy Postgres” on Amazon RDS until 2021, then sharded only because Postgres VACUUM stalls threatened transaction-ID wraparound¹⁰.

flowchart TD
    S[System slow or over capacity?] --> P[Profile the bottleneck]
    P --> Q{Missing index or bad query?}
    Q -->|Yes| F1[Fix the query]
    Q -->|No| RW{Read-bound or write-bound?}
    RW -->|Reads| C[Add cache / read replicas / CDN]
    RW -->|Writes| V{Single node still viable?}
    V -->|Yes| V1[Vertical bump on primary]
    V -->|No| VP{Vertical partition possible?}
    VP -->|Yes| V2[Split table groups per DB]
    VP -->|No| SH[Shard by domain key]

When scaling is the wrong answer

Sometimes the right move is to make the problem smaller, not the infrastructure bigger.

• Fix the query. A missing index turns a 100 ms query into a 10-second one. Adding replicas multiplies the slow query, not the fast one. Run EXPLAIN ANALYZE before you run Terraform.
• Fix the algorithm. An O(n^2) loop through 1 million items is 10^12 operations. O(n log n) is 2 * 10^7. No amount of hardware closes a 50,000x gap.
• Fix the access pattern. The N+1 query problem (100 ORM queries where one join would do) is not solved by a bigger database.
• Shed load. Reject requests you cannot serve on time rather than queueing them until everything falls over. AWS’s Builders’ Library prescribes load shedding as the first line of defense against overload, because retries stack up and create a positive feedback loop that amplifies the problem¹⁴.
• Question the cloud bill. 37signals spent $3.2 million per year on AWS in 2022¹⁵. Their cloud exit saved nearly $2 million per year by 2024¹⁶. If your workload is stable and predictable, the cheapest scaling move might be buying hardware.

Real-World Example

Figma: from one Postgres instance to horizontal sharding

Figma’s database traffic grew approximately 3x annually, and their database stack grew almost 100x between 2020 and 2024⁶¹⁷. They did not shard on day one. They exhausted every simpler option first.

Phase 1: Vertical scaling (2020). A single RDS Postgres instance on AWS’s largest physical instance. CPU utilization hit 65% at peak¹⁷. No code changes needed.

Phase 2: Vertical partitioning (2022). Groups of related tables (“Figma files,” “Organizations,” “Comments”) moved onto dedicated physical databases. They went from 1 database to 12 vertical partitions. Combined with caches and read replicas, this stack grew nearly 100x from 2020 to 2024 without touching application sharding logic⁶.

Phase 3: Horizontal sharding (2023). A Go service called DBProxy sits between the application and PgBouncer, parses SQL into an AST, extracts the shard key (UserID, FileID, or OrgID), and routes to the correct physical Postgres⁶. They evaluated CockroachDB, TiDB, Spanner, and Vitess, and rejected all of them because “we would have had to rebuild our domain expertise from scratch” under aggressive timeline pressure⁶.

Key engineering decisions:

• No unified shard key. Different table groups shard on different keys via “colos” (colocations of tables sharing a shard key)⁶.
• Logical shards as Postgres views. Each logical shard is a view over the physical table, so rollout is reversible at the feature-flag level⁶.
• 9 months for the first sharded table. The engineering cost of sharding is real. Figma’s first horizontally sharded table took 9 months end-to-end, with only 10 seconds of partial availability impact on primaries⁶.

The lesson: exhaust vertical scaling, then vertical partitioning, then cache, before you shard. Every stage is cheaper and lower-risk than the next.

Trade-offs

These levers are not substitutes you pick one of; they are stages applied in sequence along the scaling roadmap. Use this as a reference for when to reach for which lever, not as a menu of alternatives.

The levers above appear in the order the chapter recommends applying them: profile, then scale up, then absorb reads (cache, replicas, CDN), then scale the stateless tier out, then partition, then shard. See the decision flowchart earlier in this chapter for the same progression in diagram form.

Common Pitfalls

Exercise

Design Challenge: You run a B2B SaaS API serving 1,000 enterprise customers. Traffic is 5,000 RPS average, 20,000 RPS peak, 70% reads. Today you run one PostgreSQL RDS db.r6i.4xlarge (16 vCPU, 128 GB RAM) and four stateless API servers behind an ALB. Database CPU is at 80% at peak, API CPU at 40%. You expect 4x growth next year. Plan the scaling roadmap.

Key Takeaways

• Vertical scaling is the easy first move; horizontal scaling is the inevitable endgame for anything that keeps growing.
• Statelessness is what makes horizontal scaling cheap. Push state into a database, cache, queue, or object store, and the app tier becomes disposable.
• Read scaling and write scaling are different problems solved with different tools. Do not confuse them.
• Before scaling, check whether the real fix is a faster query, a missing cache, or a feature you could remove.
• Shopify served 489 million edge RPM during BFCM 2025 (driving $14.6 billion in 4 days) from a monolith by isolating failure domains into pods, not by decomposing into microservices¹⁹.
• Figma grew their database stack nearly 100x between 2020 and 2024 across vertical scaling, caches, and vertical partitioning before their first horizontal shard landed, and even then the first shard took 9 months⁶.
• Real systems mix all of these strategies. Purity is not the goal; availability, latency, and cost are.

Further Reading

• Stack Overflow: The Architecture, 2016 Edition - Nick Craver on why vertical scaling worked for over a decade; includes every server spec and utilization graph.
• A Pods Architecture to Allow Shopify to Scale - Xavier Denis on the pattern behind Shopify’s blast-radius isolation, and why post-sharding cross-shard failures forced them to fully isolate each shard.
• How We Prepare Shopify for BFCM (2025) - Year-round load-testing discipline with 2024/2025 scale numbers; the gold standard for capacity planning.
• How Figma’s Databases Team Lived to Tell the Scale - Sammy Steele’s 9-month horizontal-sharding writeup, including DBProxy, colos, and the decision to stay on Postgres.
• Lessons Learned from Sharding Postgres at Notion - The 480-logical-shards decision, why it divides well, and the VACUUM stalls that forced the move.
• How Discord Stores Trillions of Messages - Bo Ingram on hot partitions, Rust data services, and the Cassandra to ScyllaDB migration that cut p99 from 40-125 ms to 15 ms.
• The Twelve-Factor App: Processes - The canonical statement on stateless share-nothing processes; short enough to read in 5 minutes.
• Using Load Shedding to Avoid Overload (Amazon Builders’ Library) - David Yanacek on why overload amplifies itself and how to break the feedback loop before it cascades.
• Sharding and IDs at Instagram - The 64-bit shard-encoded primary-key pattern that lets any app server route without an external lookup.

Flashcards

References