Distributed Systems Patterns Cheat Sheet

In a leaderless or quorum-replicated system with N replicas, a write must be acknowledged by W of them and a read must contact R of them. The key inequality is R + W > N: it forces the read set and the write set to overlap by at least one replica, so any read is guaranteed to see at least one copy of the most recent successful write. Common configurations: W = N and R = 1 makes reads fast but writes slow and fragile (any replica down blocks writes); R = N and W = 1 is the reverse; W = R = (N/2) + 1 (a majority quorum) balances both and is what consensus systems use. A second rule, W > N/2, prevents two conflicting writes from both succeeding by ensuring write sets overlap. Quorums tolerate failures: with N = 3 and W = R = 2 you survive one node down. Note that overlap guarantees you read a recent value but not necessarily the absolute newest if writes are concurrent - you may need version vectors or read-repair to reconcile.

Range partitioning

Assign contiguous key ranges to shards (A-F here, G-M there). Great for range scans and ordered queries, but prone to hotspots if keys are skewed (e.g. everyone writing the latest timestamp hits one shard).

Hash partitioning

Shard by hash(key) mod N. Spreads load evenly and avoids hotspots, but destroys key locality (range scans must hit every shard) and resharding when N changes moves almost all keys.

Consistent hashing

Map both keys and nodes onto a hash ring; a key belongs to the next node clockwise. Adding or removing a node only remaps the keys in its arc, not the whole dataset - the standard fix for hash partitioning's reshard problem.

Virtual nodes

Give each physical node many points on the ring so load is smoother and rebalancing on node changes is finer-grained. Used by Dynamo/Cassandra-style systems.

Directory / lookup-based

Keep an explicit map from key (or key-range) to shard in a lookup service. Most flexible (arbitrary placement, easy rebalancing) but the directory is an extra hop and a potential single point of failure.

Consensus is getting a set of nodes to agree on a single value (or an ordered log of values) despite crashes and message delays. It underpins leader election, distributed locks, configuration, and replicated state machines. Paxos is the classic algorithm - provably correct but famously hard to understand and implement. Raft was designed to be understandable: it elects a single leader for a term, the leader appends entries to a replicated log and commits an entry once a majority of followers have acknowledged it, and followers redirect writes to the leader. If the leader fails, followers notice a missing heartbeat, increment the term, and hold an election; a candidate wins by collecting votes from a majority. Both rely on majority quorums, so a cluster of 2F + 1 nodes tolerates F failures (5 nodes survive 2 down) - which is why consensus clusters are sized at odd numbers like 3 or 5. The practical takeaway for interviews: do not roll your own consensus; lean on Raft-based systems (etcd, Consul, ZooKeeper's ZAB variant) to elect leaders and store critical metadata.

Outbox pattern

Write the business row and an 'event to publish' row in the same local DB transaction; a separate relay reads the outbox and publishes to the broker. Solves the dual-write problem (DB committed but message lost) and gives at-least-once publishing.

Change data capture (CDC)

Stream a database's commit/replication log (e.g. via Debezium) as a feed of change events. Lets downstream systems, caches, and search indexes react to data changes without the source service publishing them explicitly.

Circuit breaker

Wrap calls to a dependency; after a failure threshold the breaker OPENS and fails fast (no call) for a cooldown, then HALF-OPENS to test recovery. Prevents hammering a sick dependency and cascading failures.

Bulkhead

Isolate resources (thread pools, connection pools) per dependency so a failure or slowdown in one cannot exhaust capacity shared by the others - like watertight compartments in a ship.

Backpressure

Signal upstream producers to slow down when a consumer is saturated (bounded queues, flow-control, rejecting/shedding load) instead of buffering unbounded and crashing.

Idempotency key

A client-supplied unique key per logical operation; the server stores the result keyed by it so retries return the same outcome without re-applying side effects. The practical backbone of at-least-once + exactly-once-effect.

Retry with backoff + jitter

Retry transient failures with exponentially increasing delays plus randomization, so retries do not synchronize into a thundering herd. Pair with a retry budget and idempotency.

• ·The network is not reliable - messages are lost, delayed, duplicated, and reordered; a missing response means 'unknown', not 'failed'. Design for ambiguity.
• ·There is no global clock - wall-clock skew makes timestamp ordering unsafe; use logical clocks (Lamport), version vectors, or tightly-bounded clocks (TrueTime) for ordering.
• ·The Fallacies of Distributed Computing - assuming zero latency, infinite bandwidth, a secure/homogeneous network, or zero transport cost will eventually break your design.
• ·Split-brain - a partition can leave two sides each believing they are the leader and accepting writes; prevent it with quorum/fencing tokens, not just heartbeats.
• ·Cascading failure - one slow dependency backs up callers' threads/queues until the whole system falls over; contain it with timeouts, circuit breakers, and bulkheads.
• ·Thundering herd / retry storms - synchronized retries or cache expiries stampede a recovering service; use jittered backoff, request coalescing, and staggered TTLs.
• ·Stale reads after failover - a promoted follower may be behind the old leader; reads can go backwards in time unless you read from a quorum or pin to the leader.
• ·Duplicate delivery is normal - at-least-once means handlers WILL see repeats; non-idempotent side effects (charging a card twice) are a design bug, not bad luck.