What this article is for
Use this as a scaling playbook when traffic, data, or org complexity outgrows a single box. It answers: “horizontal vs vertical scaling,” “how load balancers work,” “when to add caching,” “database read replicas vs sharding,” and “what to measure before buying hardware.” Written for staff engineers, tech leads, and founders planning reliability.
The scaling challenge in one sentence
Scaling means keeping latency, error rate, and cost acceptable as load grows—without rewriting everything every quarter. It combines architecture (stateless app tiers, queues), data strategy (indexes, replicas), and process (load tests, SLOs, incident drills).
Horizontal vs vertical scaling
Vertical scaling (bigger CPU/RAM/disk) is simple until you hit hardware ceilings or cloud SKU jumps. Horizontal scaling adds more instances behind a load balancer; requires stateless application servers or careful session affinity. Most web stacks end up horizontal at the app layer with a scaled data tier.
Load balancing and health
Layer-7 HTTP balancers route by path, host, or headers; layer-4 balancers are cheaper and faster for TCP. Use active health checks (HTTP GET to /health) with sensible timeouts—drain connections before killing instances. Algorithms: round-robin, least connections, weighted, or consistent hashing for cache-friendly routing.
Caching where it pays
- CDN for static assets and cacheable GET APIs.
- HTTP cache headers for public reads; beware personalization.
- In-memory caches (Redis/Memcached) for hot keys—always define TTL and invalidation.
- Application-level memoization for expensive pure computations.
Async work and backpressure
Move non-user-critical work to queues (email, reports, webhooks). Producers must respect backpressure—if consumers lag, shed load or scale consumers instead of unbounded memory growth.
Database scaling path
Indexes and query discipline
Before replicas: fix N+1 queries, add missing indexes, and cap expensive analytics on OLTP primaries.
Read replicas
Offload read-heavy dashboards; accept replication lag—design UX for eventual consistency or route critical reads to primary.
Sharding / partitioning
Split data by a shard key (tenant id, user id). Cross-shard queries become painful—only shard when replicas and vertical scale are exhausted.
Capacity planning and SLOs
Define SLOs (e.g. p95 API < 300ms). Load test at 2× expected peak; watch saturation (CPU, connections, thread pools, GC). Autoscale on metrics that correlate with user pain, not only CPU.
FAQ
What breaks first in real systems?
Often the database or a single shared dependency (auth, payment API)—not the web servers.
Is microservices required to scale?
No. Well-factored modular monoliths scale far; extract services when team or failure isolation demands it.
Key takeaways
- Scale stateless tiers horizontally; treat data as the hardest part.
- Use caching and queues deliberately with TTLs and backpressure.
- Measure with SLOs and load tests, not intuition.
