Scalability Patterns
Scalability is a system's ability to handle growth — more users, more data, more traffic — without falling over or requiring a rewrite. The patterns are well-established, and most come down to a few moves: make services stateless so you can add more of them, cache aggressively at every layer, and recognize that the database is usually the bottleneck.
The foundational choice is how you scale: up (a bigger machine) or out (more machines). Vertical scaling is simpler but hits a hard ceiling and stays a single point of failure; horizontal scaling has no ceiling and adds redundancy, at the cost of managing state and coordination. For most growth, scaling out wins.
TL;DR
- Horizontal scaling > vertical for most use cases.
- Stateless services enable easy scaling.
- Cache aggressively at every layer.
- The database is often the bottleneck — plan replicas/sharding.
Quick Example
Externalize session state so any server can handle any request:
Core Concepts
Scaling dimensions
- Vertical (scale up) — add CPU/RAM/disk to one server. Simple, no code changes, but hits hardware limits and remains a single point of failure. Good as a quick fix and for database primaries.
- Horizontal (scale out) — add more servers. No hard ceiling and built-in redundancy, but requires statelessness and coordination. The default for web/app tiers.
CAP theorem
In a distributed system you can guarantee only two of three under a network partition:
Since partitions are unavoidable in distributed systems, the real choice is usually consistency vs availability during a partition.
Stateless Services & Load Balancing
Stateless services are the precondition for horizontal scaling — push state (sessions, uploads) to shared stores, then put a load balancer in front:
Common algorithms: round robin (even), least connections (send to least busy), IP hash (sticky — same client → same server), weighted (favor bigger servers).
Caching Layers
Cache at every level so requests resolve as close to the user as possible:
See Caching.
Database Scaling
Read replicas — scale reads
Write to the primary, read from replicas. Watch for replication lag (a read right after a write may be stale).
Sharding — scale writes and data size
Sharding removes the single-database ceiling but complicates cross-shard queries and transactions — adopt it only when replicas aren't enough. See Database Replication.
Async Processing & CDNs
Move slow work off the request path with a queue + worker pool — it decouples slow operations, absorbs spikes, and enables retries:
And serve static assets (images, CSS/JS, cacheable API responses) from a CDN edge near the user, falling back to origin on a miss.
Identifying Bottlenecks
Best Practices
- Make services stateless — externalize sessions and uploads.
- Scale out, not just up — and remove single points of failure.
- Cache at every layer with deliberate TTLs and invalidation.
- Scale reads with replicas first, shard only when you must.
- Push slow work async and serve static assets from a CDN.
Common Mistakes
In-memory state on a scaled-out tier
Sharding too early
FAQ
Should I scale vertically or horizontally?
Start vertical for simplicity (a bigger box, no code changes) until you hit cost or hardware limits or need redundancy — then scale horizontally. Stateless tiers (web/app) should scale out by default for resilience and headroom. Databases are often scaled vertically for the primary plus horizontally via read replicas, and sharded only when necessary. Most real systems use both dimensions.
What does the CAP theorem mean for my design?
Under a network partition you can keep either consistency or availability, not both. So decide per use case: payments and inventory want consistency (reject rather than serve stale/conflicting data), while feeds and analytics can favor availability (serve possibly-stale data and reconcile later). Many systems mix both — strong consistency where correctness is critical, eventual consistency where availability matters more.
How do read replicas differ from sharding?
Read replicas copy the whole dataset to additional nodes to scale read throughput — writes still go to one primary, and you must tolerate replication lag. Sharding splits the data itself across nodes by a shard key to scale writes and total data size — each shard holds a subset. Use replicas first (simpler); reach for sharding when write volume or data size exceeds a single primary.
Where do most scalability bottlenecks appear?
Usually the database — slow queries, missing indexes, N+1 access patterns, and write contention — followed by synchronous processing of slow work on the request path. The fixes in order: add indexes and fix queries, cache hot reads, move slow work to async queues, add read replicas, and only then shard. Profile with p95/p99 latency and resource metrics before optimizing.
Related Topics
- Caching — Caching strategies
- System Design — Putting patterns together
- Database Replication — Replicas and sharding
- Message Queues — Async processing
- Cloud Networking — Load balancing