System design is deciding how the parts of a software system fit together so it stays fast, reliable and affordable as usage grows. The core building blocks are load balancers, stateless application servers, databases with replication and sharding, caches, message queues, CDNs and monitoring. Good design starts from requirements and trade-offs, not from a list of fashionable technologies.

Why does system design matter?

A simple app with one server and one database works well for hundreds of users. At hundreds of thousands, new problems appear: the server runs out of CPU, the database becomes the bottleneck, a single failure takes everything down, and pages load slowly for users far from the data centre. System design is the toolkit for these problems. It's also how teams avoid over-engineering: knowing what each tool solves lets you add complexity only when you need it.

How should you start a design?

Always clarify requirements first:

  • Functional requirements: what must the system do? (Post messages, follow users, show a feed.)
  • Non-functional requirements: how fast, how available, how consistent, how secure?
  • Scale estimates: daily users, requests per second, data stored per day, read-to-write ratio.

A quick estimate: 1 million daily users, each making 50 requests a day, is 50 million requests a day, about 580 per second on average, with peaks perhaps several times that. Such back-of-the-envelope numbers tell you whether one database is enough or whether you need caching and replicas.

What are the core building blocks?

ComponentWhat it doesSolves
Load balancerSpreads requests across serversSingle-server limits, failures
Stateless app serversHandle requests without storing session data locallyEasy horizontal scaling
Cache (e.g. Redis)Keeps frequently read data in memoryDatabase load, latency
Database replicationCopies data to read replicasRead scaling, failover
ShardingSplits data across databases by a keyWrite scaling, very large datasets
Message queueBuffers work for background processingSpikes, slow tasks, decoupling
CDNServes static files from locations near usersLatency, bandwidth
Monitoring and logsMeasures health and errorsKnowing when and why things break

Vertical or horizontal scaling?

Vertical scaling means a bigger machine: simple, but limited and a single point of failure. Horizontal scaling means more machines behind a load balancer: nearly unlimited and resilient, but it requires stateless servers (sessions stored in a shared store or tokens) and careful data design. Most systems scale vertically first, because it's cheaper in engineering time, then horizontally when needed.

How does caching help, and what can go wrong?

Reading from memory is much faster than reading from a database on disk, so caching popular data (a celebrity's profile, a product page, a computed feed) removes most of the load. The hard parts are invalidation (when data changes, stale cache entries must be updated or expire), cache stampedes (many requests rebuilding the same expired entry at once) and deciding what to cache. A common pattern is cache-aside: check the cache, fall back to the database on a miss, then store the result with an expiry time.

Database reads per 1,000 requests at different cache hit rates
No cache1,000
80% hit rate200
95% hit rate50

Arithmetic: misses = requests × (1 − hit rate).

How do databases scale?

  • Indexes first: most slow queries are fixed by the right index. See indexing explained.
  • Read replicas: send reads to copies; writes go to the primary. Replicas may lag slightly behind.
  • Sharding: split users across databases by user ID, for example. Queries across shards become harder.
  • Choosing the right database: relational for transactions and relationships, document or key-value stores for flexible or very high-volume data. See MongoDB vs PostgreSQL.

What is the CAP theorem?

In a distributed database, when a network partition separates nodes, you must choose between consistency (every read sees the latest write) and availability (every request gets a response). Payment systems usually favour consistency; social feeds often favour availability and accept that a like count may be slightly stale. The practical lesson: decide, per feature, how fresh data must be.

What does a design for a chat app look like?

  1. Clients connect over WebSockets to chat servers behind a load balancer.
  2. Chat servers are stateless apart from open connections; a shared store records which server each user is connected to.
  3. Messages are written to a database partitioned by conversation ID, and published to a queue or pub/sub system.
  4. Delivery: the server holding the recipient's connection pushes the message; if they're offline, a push notification is sent.
  5. Media goes to object storage and is served through a CDN.
  6. Monitoring tracks delivery latency and failed sends.

How do queues make systems more reliable?

Some work doesn't need to happen while the user waits: sending emails, generating PDFs, resizing images, running an AI model on an upload. Putting these jobs on a queue lets the web request finish quickly while worker processes handle the backlog at their own pace. Queues also absorb traffic spikes and let a failing job be retried without losing it. The trade-off is eventual completion rather than immediate results, so design the user experience accordingly ("we'll email your receipt shortly").

How should you approach a design interview?

  1. Clarify requirements and constraints (5 minutes).
  2. Estimate scale roughly.
  3. Draw a simple high-level design first.
  4. Identify bottlenecks and single points of failure.
  5. Deepen one or two components (data model, caching, scaling).
  6. State trade-offs and what you'd monitor.

Interviewers care more about structured thinking and honest trade-offs than about naming every technology.

How do you design for failure?

At scale, something is always failing: a disk, a server, a network link, a third-party API. Good designs assume this. Run at least two instances of every service behind a load balancer with health checks. Replicate databases and practise failover. Put timeouts and retries (with backoff) on every network call, and circuit breakers so one slow dependency doesn't stall everything. Make background jobs idempotent so a retried job doesn't charge a customer twice. Keep backups and test restoring them. Reliability comes far more from these habits than from any single technology.

Which numbers should every engineer know?

Rough latencies help you reason about designs: reading from memory takes nanoseconds, a fast SSD read takes tens to hundreds of microseconds, a round trip within a data centre takes around half a millisecond, and a round trip between continents takes well over a hundred milliseconds. The practical lessons follow directly: cache hot data in memory, avoid many sequential network calls in one request, and put content close to users with a CDN. For users across India, choosing a cloud region in India noticeably reduces latency compared with distant regions.

Measure before optimising: a simple latency breakdown of one slow request usually shows exactly which of these costs dominates. Often it's a missing index or a loop making one database call per item, problems no amount of extra servers would fix.

How do you learn system design?

Build first, then design. Make a full-stack app, deploy it, add caching when a page is slow, add a queue for emails, and read engineering blogs about how real systems evolved. Program Zero's full-stack course covers React, Node.js APIs, authentication, WebSockets and system design (caching, load balancing, feed and chat architectures), and the final capstone includes a "how this scales to production" design document. Start with building a REST API with Node.js.