Top 50 Distributed Systems Interview Questions 2026

Top 50 Distributed Systems Interview Questions 2026

Distributed systems questions show up in backend, infrastructure, SRE, data platform and staff-level loops, and they rarely arrive as "design Twitter." They arrive as "your leader just paused for eight seconds - what breaks?" or "the client retried and the customer was charged twice - why?" The vocabulary (CAP, quorums, Raft, vector clocks) is the entry fee. What the interviewer is actually testing is whether you reason about partial failure by default: messages that are delayed rather than lost, nodes that are slow rather than dead, clocks that drift, and retries that turn a blip into an outage.

This is the theory-and-failure-modes companion to the system design listicles. The questions escalate from foundations through consistency, replication, consensus, clocks, partitioning and transactions, then failure handling, and finish with five scenario questions in the shape live rounds use.

Practice alongside this post. The system design question bank drills the same concepts as timed multiple-choice, and the databases question bank covers replication and isolation. Keep the distributed systems patterns cheat sheet and the system design tradeoffs cheat sheet open while you read, plus the concurrency essentials cheat sheet for the single-machine version of the same problems. The walkthroughs on consensus and distributed transactions go deeper on Parts 4 and 6. Sibling posts: Top 30 system design interview questions, Top 50 microservices interview questions and Top 50 Kafka interview questions.

Part 1: Foundations (1-7)

1. What makes a system "distributed," and why is it harder than concurrency on one machine?

A distributed system is a set of processes that coordinate only by sending messages, with no shared memory and no shared clock. On one machine a crash takes everything down together. Across machines you get partial failure: one node is down, slow or partitioned while the rest keep running, and you cannot tell which. A request that times out may have failed, may have succeeded with the reply lost, or may still be in flight.

Name the three root causes: unreliable networks (loss, delay, reordering, duplication), unreliable clocks (drift, jumps, no global "now"), and process pauses (GC, VM migration) that make a live node look dead. Almost every question below follows from those three.

2. What are the fallacies of distributed computing?

The eight fallacies, from Peter Deutsch and colleagues at Sun: the network is reliable; latency is zero; bandwidth is infinite; the network is secure; topology doesn't change; there is one administrator; transport cost is zero; the network is homogeneous.

Don't just recite them. The interviewer wants one concrete bug per fallacy. "Latency is zero" produces chatty N+1 service calls that are fine in one zone and fall over across regions. "The network is reliable" produces code with no timeout on an outbound call.

3. What does the CAP theorem actually say?

During a network partition, a system must choose between consistency (here meaning linearizability - every read sees the latest write) and availability (every request to a non-failed node gets a non-error response). Brewer stated it as a conjecture in 2000; Gilbert and Lynch proved a formal version in 2002.

CAP describes behaviour during a partition. When the network is healthy you can have both. A CP system (etcd, ZooKeeper) refuses writes on the minority side of a partition. An AP system (a Dynamo-style store) accepts writes on both sides and reconciles later.

4. Why is "pick two of three" a misleading way to describe CAP?

Because partitions are not optional. You cannot choose "CA" for a system that runs on a network; the network will partition anyway. The real choice is C or A when P happens.

The definitions are also narrow: CAP's consistency is linearizability only, and its availability is total. Most real systems are neither, and they tune per operation. A strong answer: "CAP is a useful boundary, but I'd rather talk about which operations need linearizability and what latency I pay for it."

5. What is PACELC and why do interviewers prefer it?

PACELC (Daniel Abadi) extends CAP: if Partition, choose Availability or Consistency; Else, choose Latency or Consistency. The "else" half matters day to day, because partitions are rare but every request pays the latency cost of coordination.

Follow-up: "where does your design sit?" Answer per operation - a balance check might be PC/EC while a "likes" counter is PA/EL.

6. What are FLP impossibility and the Two Generals problem, and why should you care?

Two Generals: two parties on a lossy channel can never be certain they agree, because the last acknowledgment can always be lost. It is why no protocol gives "exactly-once delivery" over an unreliable network.

FLP (Fischer, Lynch, Paterson, 1985): in a fully asynchronous system, no deterministic algorithm can guarantee consensus terminates if even one process may crash, because a crashed process is indistinguishable from a slow one.

Why it matters: Raft and Paxos work around FLP by assuming partial synchrony. They are always safe (never decide two values) but only live when timing is good enough to elect a stable leader. That is why a flapping network can stall a Raft cluster without corrupting it.

7. What failure models do you design for?

• Crash-stop: a node works correctly until it halts forever.
• Crash-recovery: a node crashes and comes back, possibly without its in-memory state. This is the realistic datacenter model, and why consensus protocols persist their term and vote before replying.
• Byzantine: a node can behave arbitrarily, including lying.

Tolerating f crash failures in a consensus group needs 2f + 1 nodes; tolerating f Byzantine failures needs 3f + 1. Inside one company you almost always assume non-Byzantine nodes and catch corruption with checksums. Saying "I'm assuming crash-recovery, non-Byzantine" out loud is a strong opening move.

Part 2: Consistency models (8-13)

8. What is linearizability?

Linearizability makes a replicated object behave as if there were a single copy and every operation took effect atomically at some instant between its start and its end. Once any read returns a new value, every later read (in real time) returns that value or a newer one.

You need it for leader election, locks and uniqueness constraints ("this username is taken") - anything where clients must agree on one winner. The cost is coordination: a write needs a quorum round trip, and a read needs a quorum or a leader that has confirmed it is still leader.

Bonus point: linearizability is composable - if each object is linearizable, the whole system is.

9. How does sequential consistency differ from linearizability?

Sequential consistency (Lamport, 1979) requires all operations to appear in some single total order that respects each client's own program order. That order need not match real time.

Example: client A writes x = 1 and gets an acknowledgment. Afterwards, client B reads x and gets 0. That is legal under sequential consistency (B's read can be ordered first) but illegal under linearizability, because the write had already completed.

In short: linearizability = sequential consistency + real-time ordering. "Everyone agrees on an order" is weaker than "the order matches the clock on the wall."

10. What is causal consistency?

Operations that are causally related are seen by everyone in the same order; concurrent operations may be seen in different orders on different nodes.

Example: Alice posts "I'm leaving the group," then "the group is terrible." Without causal consistency, Bob might see the second post without the first. With it, anyone who sees a reply also sees what it replied to.

The point to make: causal consistency is about the strongest model a system can offer while staying available during a partition. It is implemented by tracking dependencies (version vectors or explicit dependency metadata) and delaying visibility of a write until its dependencies are visible.

11. What does "eventual consistency" promise, and what doesn't it?

Only that if writes stop, replicas eventually converge. It says nothing about how long that takes, what you read meanwhile, or which value wins. It does not promise you see your own writes or that reads move forward in time, and with last-writer-wins it silently discards one of two concurrent updates.

The follow-up is "how do you make it usable?" Add session guarantees (next question), choose a deliberate conflict-resolution strategy, and measure replication lag so "eventually" has a number on a dashboard.

12. What are read-your-writes and the other session guarantees?

Session guarantees (from the Bayou work by Terry et al.) are per-client promises on top of an eventually consistent store:

Implementations: route a user's reads to the leader for a short window after they write, pin a session to one replica, or have the client carry its last-seen log position and only read from replicas that have caught up to it.

13. What is the difference between linearizability and serializability?

• Serializability is a transaction isolation property: concurrent multi-object transactions produce the same result as some serial order. It says nothing about real time.
• Linearizability is a recency property of single operations on single objects.

A serializable database can legally serve you a stale snapshot; a linearizable register gives no multi-key atomicity. Both together - serializable transactions ordered consistently with real time - is strict serializability, which Spanner calls external consistency.

Interviewers ask this because candidates use "consistent" loosely. Precision about which property each operation needs is the signal.

Part 3: Replication (14-21)

14. What are the trade-offs between synchronous and asynchronous replication?

In single-leader replication, writes go to the leader, which streams a log to followers.

• Synchronous: the leader waits for a follower before acknowledging. No acknowledged write is lost on leader failure, but every write pays a round trip and a slow follower stalls writes.
• Asynchronous: acknowledge immediately. Fast, but a leader failure can lose acknowledged writes.
• Semi-synchronous: wait for one follower or a quorum. The common compromise.

Follow-up: "the leader died under async replication - what happened to the lost writes?" They are gone, and if the old leader rejoins and replays them you can get conflicts such as reused auto-increment IDs. Failover tooling must discard them deliberately.

15. What problems does replication lag cause?

With async followers serving reads you get stale reads after your own write (violating read-your-writes), time going backwards when consecutive reads hit replicas with different lag (violating monotonic reads), and causality violations such as an answer visible before its question.

Mitigations: read from the leader for data the user just modified; sticky routing by user ID; carry a version token from the write and redirect if a replica is behind it; and monitor lag as an SLO, draining replicas that fall too far behind from the read pool.

16. When would you use multi-leader replication, and how do you resolve conflicts?

Multi-leader (each region or device accepts writes) suits multi-region write latency, offline-capable clients and collaborative editing. The cost is conflicts:

• Last-writer-wins: highest timestamp wins. Simple, silently loses data, and vulnerable to clock skew.
• Merge by data type: union sets, sum counters - CRDTs (question 21).
• Keep siblings: store both versions and let the application or user resolve.
• Avoid conflicts: give each record a "home" leader so conflicts only arise on failover.

Interviewers like the last option: avoiding a hard problem beats solving it. Also note that with all-to-all topologies writes can arrive out of causal order, so you detect concurrency with version vectors, not arrival order.

17. How does leaderless (Dynamo-style) replication work?

There is no leader. A client or coordinator sends each write to all N replicas for the key and treats it as successful once W acknowledge; reads query replicas, wait for R responses and return the newest version. Amazon's Dynamo paper popularized the design; Cassandra and Riak follow it.

Benefits: no failover, tolerance of slow nodes, per-request tunable consistency. Costs: conflicts are possible, so you need versioning, and repairing divergent replicas is your job (question 20). Follow-up: "how do you know which version is newest?" Timestamps (with clock risk) or version vectors, which can detect concurrent writes and keep both.

18. What does R + W > N guarantee, and what doesn't it?

If W + R > N, every read quorum overlaps every write quorum in at least one node, so a read should reach a replica holding the latest acknowledged write. A common setting is N = 3, W = 2, R = 2.

What it does not guarantee - the real question:

• Linearizability. An in-progress write can be seen by one reader and missed by a later one, unless reads perform read repair before returning.
• Overlap under sloppy quorums (next question).
• Conflict freedom. Concurrent writes still conflict; you just see at least one.
• Rollback. A write that reached fewer than W nodes is not undone on the nodes that accepted it.

19. What are sloppy quorums and hinted handoff?

When some of a key's home replicas are unreachable, a sloppy quorum accepts the write on other healthy nodes so it still collects W acknowledgments. Those stand-ins keep a hint and forward the data to the proper replica when it recovers (hinted handoff). Dynamo used this for write availability.

The cost: W + R > N no longer implies overlap, because your read quorum may be home replicas that haven't received the write yet. Name it explicitly - an availability-over-consistency choice hiding inside something that sounds like a consistency mechanism.

20. How do replicas converge after divergence?

• Read repair: when a read sees different versions, the coordinator writes the newest back to the stale replicas. Cheap, but only fixes data that gets read.
• Anti-entropy: a background process compares replicas using Merkle trees (hash trees over key ranges). Compare roots, recurse only into subtrees whose hashes differ, and transfer only the divergent ranges.

Follow-up: "what about deletes?" A delete is written as a tombstone so anti-entropy doesn't resurrect the value from a replica that missed it. Tombstones must outlive the repair interval, which is why they are a common cause of bloat and slow reads in Cassandra-style systems.

21. What is a CRDT and when would you use one?

A conflict-free replicated data type has a merge that is commutative, associative and idempotent, so replicas that receive the same updates in any order, any number of times, converge with no coordination.

• G-Counter: one count per replica; value is the sum; merge takes per-replica max.
• PN-Counter: two G-Counters, for increments and decrements.
• OR-Set: adds are uniquely tagged, so a concurrent add and remove resolve as "add wins."
• LWW-Register: last writer wins by timestamp.
• Sequence CRDTs for collaborative text.

Use them for multi-region counters, presence, carts and collaborative documents. The limit to mention: CRDTs cannot enforce invariants like "balance never below zero" or "one owner per username." Those need consensus.

Part 4: Consensus, leader election and fencing (22-29)

22. What problem does consensus solve, and where do you meet it?

Consensus gets nodes to agree on a value - in practice, a replicated log - such that agreement is never violated when nodes crash or messages are lost, and progress continues while a majority can communicate.

You meet it in leader election, locks, configuration and metadata stores, and strongly consistent databases. etcd (and so Kubernetes' cluster state) and Consul use Raft; ZooKeeper uses its own protocol, ZAB; Kafka's KRaft mode uses a Raft-based protocol for metadata; Google's Chubby and Spanner use Paxos.

The advice interviewers want: don't implement consensus yourself. Use one of these as a coordination service and keep the group small.

23. Walk through how Raft works.

Raft (Ongaro and Ousterhout, 2014) has three parts:

1. Leader election: nodes are followers, candidates or leaders, and time is divided into numbered terms. A follower that hears no heartbeat within a randomized election timeout (the paper suggests something like 150-300 ms) increments the term, becomes a candidate and requests votes. Each node votes at most once per term; a majority makes a leader.
2. Log replication: the leader appends commands to its log and sends AppendEntries. Once an entry is stored on a majority it is committed and applied to the state machine.
3. Safety: a node only votes for a candidate whose log is at least as up to date as its own (by last term, then last index), so every elected leader already holds every committed entry.

Mention the randomized timeouts: they make split votes rare and short.

24. In Raft, why can't a new leader commit an old-term entry just by counting replicas?

Because an entry from an earlier term can sit on a majority and still be overwritten. In the scenario shown in Figure 8 of the Raft paper, a leader replicates an entry partway and crashes, and a later leader whose log wins the up-to-date comparison replaces it - even after it reached a majority.

Raft's rule: a leader only commits entries from its current term by counting replicas. Once a current-term entry commits, all earlier entries commit with it through the log matching property. That is why new leaders commonly append a no-op entry at the start of their term.

This question separates people who read a summary from people who read the paper. You need the rule and the reason, not the full trace.

25. Explain Paxos at interview depth.

Single-decree Paxos has proposers, acceptors and learners, in two phases:

1. Prepare/Promise: a proposer picks a number n higher than any it has used and sends prepare(n). Each acceptor promises to ignore proposals below n and returns the highest-numbered proposal it has already accepted, if any.
2. Accept/Accepted: with promises from a majority, the proposer sends accept(n, v), where v must be the value from the highest-numbered accepted proposal reported (or its own if none). Acceptors accept unless they have promised a higher number. A value accepted by a majority is chosen.

The "adopt the previously accepted value" rule is what makes it safe. Multi-Paxos keeps a stable leader and skips phase 1 for later slots, so in steady state it looks much like Raft. Raft was designed to be easier to understand and pins down details (log structure, elections, membership changes) that Paxos leaves open.

26. Why do consensus clusters have 3 or 5 nodes, not 4 or 6?

A cluster of N needs a majority, floor(N/2) + 1, so it tolerates floor((N-1)/2) failures:

An even count adds cost and replication latency without adding fault tolerance, and bigger groups slow writes because the leader waits on more followers. Five is common when you want to survive a planned and an unplanned outage at once. Placement matters as much as count - see scenario 50.

27. How do you serve linearizable reads from a Raft cluster?

A leader can't just answer from local state: it may have been deposed by a newer leader it hasn't heard from. Options:

• Log the read like a write. Correct, slow.
• ReadIndex: record the commit index, confirm leadership with a round of heartbeat acknowledgments from a majority, wait until applied up to that index, then answer locally.
• Leader leases: after a successful heartbeat round, assume leadership for a lease period and serve reads with no round trip. Fastest, but it depends on bounded clock drift - if clocks misbehave, two nodes can both think they hold the lease.

Followers can also serve linearizable reads by asking the leader for the read index and waiting until they have applied that far.

28. How do you implement leader election for your own service?

Delegate to a coordination service instead of running consensus in your app:

• etcd: in a transaction, create a key bound to a lease with a TTL only if the key doesn't exist; keep the lease alive. If the holder dies, the lease expires and the key disappears.
• ZooKeeper: each candidate creates an ephemeral sequential node; the lowest sequence number leads; each other candidate watches only the node just ahead of it, avoiding a thundering herd.
• A lease row in a strongly consistent database: UPDATE leader SET owner = :me, expires = now() + interval '15 seconds' WHERE expires < now() OR owner = :me.

The catch, and the next question: a lease tells you that you were leader when you checked. It doesn't stop a paused former leader acting on a stale belief.

29. What is a fencing token and why do leases need one?

The failure: client A takes a lock with a 10-second lease, then hits a long GC pause. The lease expires, client B takes the lock and writes. A wakes, still believing it holds the lock, and writes too.

The fix: the lock service issues a monotonically increasing token with each grant (A gets 33, B gets 34). Every write carries its token, and the resource rejects any token lower than the highest it has seen, so A's late write is refused.

storage on write(req):
  if req.token < max_seen_token: reject
  else: max_seen_token = req.token; apply

ZooKeeper's zxid and etcd revisions can serve as tokens. The point to land: safety is enforced at the resource, because no client-side check survives an arbitrary pause. This is the core of Martin Kleppmann's well-known critique of Redis Redlock for correctness-critical locking.

Part 5: Time and clocks (30-34)

30. Why can't you trust wall-clock timestamps to order events across machines?

Quartz clocks drift and NTP corrects them by slewing or stepping, so clocks across a fleet disagree - usually slightly, but without a hard bound in bad cases. Clocks can jump backwards, VMs pause, and leap seconds have caused real incidents.

Consequences: last-writer-wins can discard the write that actually happened later, leases checked on skewed clocks can overlap, and measured durations can go negative.

Two rules: measure durations and timeouts with a monotonic clock (CLOCK_MONOTONIC, Java's System.nanoTime()), never wall time. For ordering across machines, use logical clocks or a clock with an explicit uncertainty bound like TrueTime.

31. How do Lamport clocks work, and what is their limitation?

Each process keeps a counter, increments it on each event, attaches it to messages, and on receipt jumps forward:

on local event:   L = L + 1
on send:          L = L + 1; send(msg, L)
on receive(m, t): L = max(L, t) + 1

Guarantee: if a happened-before b, then L(a) < L(b). Breaking ties by node ID gives a total order consistent with causality.

Limitation: the converse fails. L(a) < L(b) does not mean a caused b; they may be concurrent. Lamport clocks can order events but cannot detect concurrency, which is exactly what conflict detection needs.

32. How do vector clocks detect concurrent events?

Each node keeps one counter per node. It increments its own entry on each event, sends the vector with messages, and on receipt takes the element-wise max and increments its own entry.

Comparing V and W: if V <= W element-wise (and they differ), V happened before W. If neither dominates, the events are concurrent - a genuine conflict. That is how Dynamo-style stores keep siblings instead of silently dropping one.

The cost is size, which grows with the number of participants. Systems bound it by tracking replicas rather than clients (version vectors, or dotted version vectors in Riak) and pruning old entries.

33. What is a hybrid logical clock (HLC)?

An HLC (Kulkarni et al., 2014) pairs a physical timestamp with a logical counter. The physical part tracks the largest wall-clock time seen; the counter breaks ties and preserves causality when physical clocks are behind or equal.

You get timestamps that respect happened-before like a Lamport clock, stay close to real time so "read as of time T" snapshots make sense, and have a fixed size, unlike vector clocks. CockroachDB uses HLCs for transaction timestamps.

Follow-up: an HLC doesn't bound clock uncertainty, so on its own it can't give linearizability. CockroachDB adds an uncertainty interval and restarts reads that hit values inside it.

34. How does Google Spanner's TrueTime provide external consistency?

TrueTime returns time as an interval, TT.now() = [earliest, latest], backed by GPS receivers and atomic clocks in each datacenter so the uncertainty stays within a few milliseconds.

When a transaction commits at timestamp s, Spanner performs commit wait: it doesn't make the commit visible until TT.now().earliest > s, so s is in the past on every clock. Any transaction starting after this one finishes gets a larger timestamp, which yields external consistency (strict serializability).

The interview point: Spanner converts clock uncertainty into latency - the tighter the bound, the shorter the wait. Without that time infrastructure, systems use HLCs with uncertainty windows instead.

Part 6: Partitioning and distributed transactions (35-41)

35. What are the main partitioning strategies?

Expect follow-ups on secondary indexes (local, so queries scatter-gather, versus global, so writes become distributed) and rebalancing: never hash mod N with a changing N, because nearly every key moves. More in database sharding explained.

36. How does consistent hashing work, and why add virtual nodes?

Hash nodes and keys onto the same ring; each key belongs to the first node clockwise. When a node joins or leaves, only the keys between it and its neighbour move - roughly 1/N of the data, not nearly all of it.

Plain consistent hashing gives uneven arcs, and a departing node dumps its whole load on one neighbour. Virtual nodes fix both: each physical node owns many ring points, so load averages out and a failure spreads across many nodes. Varying the count per node handles uneven hardware.

Alternatives worth naming: rendezvous (highest random weight) hashing and jump consistent hash. The consistent hashing explainer walks the ring step by step.

37. How do you handle a hot key or hot partition?

Hashing spreads keys evenly, not load. One celebrity account can swamp its partition under any scheme. In rough order of preference:

1. Cache the hot read path, with request coalescing so a miss triggers one backend call, not thousands.
2. Split the key: write to item:42#0..#15 and aggregate on read. Ideal for counters.
3. Replicate hot read-only data to more nodes.
4. Split or move the partition; many stores split ranges automatically.
5. Rate-limit or shed the pathological cases at the edge.

Follow-up: "how do you detect it?" Per-partition metrics and top-K key sampling - averages hide hot spots completely.

38. How does two-phase commit work, and why is it called blocking?

1. Prepare: the coordinator asks each participant to prepare. A "yes" vote durably promises the participant can commit, keeps its locks held, and gives up the right to abort on its own.
2. Commit/abort: if all vote yes, the coordinator logs the decision and tells everyone to commit; otherwise abort.

If the coordinator crashes after the votes but before announcing the decision, participants are in doubt: they can't commit (someone may have voted no) or abort (the decision may have been commit), so they hold locks until it recovers.

Follow-ups: 2PC needs every participant, not a majority, so it is atomic commit, not consensus. Spanner runs 2PC across participant groups that are each Paxos-replicated, so a participant can't simply vanish. Three-phase commit assumes bounded delays and is rarely used.

39. What is a saga, and orchestration versus choreography?

A saga replaces one distributed transaction with a sequence of local transactions, each with a compensating action. Book flight, book hotel, charge card; if the charge fails, cancel the hotel, then the flight.

• Orchestration: a coordinator (workflow engine or state machine) drives each step and the compensations. Easier to see and debug.
• Choreography: services react to each other's events. Looser coupling, but the flow is implicit and hard to follow past a few steps.

What interviewers probe: sagas give up isolation, so others can see intermediate states - use pending states or semantic locks. Compensations must be idempotent and retryable, and steps that can't be undone (an email sent) go last.

40. What is the transactional outbox pattern?

The dual-write problem: a service updates its database and publishes an event. A crash between the two leaves an update with no event, or an event for a change that rolled back.

The outbox: insert the event into an outbox table in the same local transaction as the business change. A relay publishes outbox rows to the broker, by polling or by change data capture from the database log (Debezium is the common tool).

BEGIN;
UPDATE orders SET status = 'PAID' WHERE id = 42;
INSERT INTO outbox (aggregate_id, type, payload)
VALUES (42, 'OrderPaid', '{"orderId":42}');
COMMIT;

Follow-up: the relay can crash after publishing but before marking the row sent, so events may go out twice. The outbox gives at-least-once delivery, and consumers must be idempotent.

41. Is exactly-once delivery possible? How do you get exactly-once effects?

Exactly-once delivery over an unreliable network isn't possible: a sender with no acknowledgment either retries (risking duplicates) or doesn't (risking loss). What systems provide is exactly-once effects: at-least-once delivery plus deduplication or idempotent application.

• Idempotency keys: the client sends a unique key per logical operation; the server stores key -> result atomically with the effect and replays the stored result on retries.
• Natural idempotency: SET balance = 100 is idempotent; balance = balance + 10 is not.
• Dedup by message ID in the consumer, in the same transaction as the side effect.

Kafka's exactly-once semantics (idempotent producers, transactions, read_committed consumers) is real but scoped to read-process-write within Kafka. Calling a payment API from the consumer still needs its own idempotency. The Kafka post covers the settings.

Part 7: Failure detection and retries (42-45)

42. How do distributed systems detect failed nodes?

They can only suspect. The basic tool is heartbeats with a timeout, which is a trade-off: short means fast detection and false positives (a GC pause triggers failover), long means slow detection.

Better approaches:

• Phi accrual failure detector (Hayashibara et al.): tracks the distribution of heartbeat inter-arrival times and outputs a suspicion level, phi, that rises as a heartbeat becomes overdue relative to normal. Cassandra and Akka use it.
• Indirect probing (SWIM, next question): ask other nodes to probe the suspect before declaring it dead, filtering out a single bad network path.

Senior follow-up: a false positive is often worse than slow detection, because failover itself is risky - rebalancing, cold caches, possible split brain.

43. What is a gossip protocol and where is it used?

Each node periodically exchanges state (membership, heartbeats, metadata) with a few random peers. Information spreads epidemically, reaching the cluster in a number of rounds that grows roughly logarithmically with its size, with no coordinator and bounded per-node load.

SWIM is the well-known membership variant: each round, ping one random member; if it doesn't answer, ask k others to ping it indirectly; only then mark it suspect, and dead after a timeout. Updates piggyback on pings. HashiCorp's memberlist and Serf (used by Consul) are SWIM-based, and Cassandra gossips cluster state.

Trade-off: gossip is eventually consistent, so it suits membership and metadata, not decisions that need one agreed answer.

44. What is split brain and how do you prevent it?

Split brain is two nodes both acting as leader and accepting writes, usually after a partition or a false-positive failure detection. Their data diverges, sometimes irreconcilably.

Prevention:

• Majority quorums: only the majority side of a partition can elect a leader; the minority stops. For two-site setups, add a witness in a third location.
• Fencing tokens at the storage layer (question 29).
• STONITH ("shoot the other node in the head"): power off or cut storage access to the old primary before taking over.
• Conservative leases: the new leader waits until the old lease must have expired.

The answer interviewers want combines two: a quorum prevents two elected leaders; fencing stops the one that doesn't know it was replaced.

45. How should clients retry, and why does jitter matter?

Retry only plausibly transient failures (timeouts, 503s, resets), only for idempotent operations or ones carrying an idempotency key, and with:

• Exponential backoff, capped.
• Jitter: without it, clients that failed together retry together, in synchronized waves. The AWS Architecture Blog's "Exponential Backoff and Jitter" analysis found "full jitter" works well.
• A retry budget: retries may add at most a small fraction of normal traffic.
• One retrying layer: three layers retrying three times each turns one request into 27 calls.
• Deadlines: propagate the remaining time budget so downstream work stops when the caller has given up.

def backoff(attempt, base=0.1, cap=10.0):
    return random.uniform(0, min(cap, base * 2 ** attempt))  # full jitter

Part 8: Scenario questions (46-50)

These are the shape most live rounds use: a symptom, a system, and "what's going on and what would you change?" Narrate hypotheses before jumping to a fix. For the framework behind these, read designing for failure: what interviewers want.

46. A dependency is failing slowly and taking your service down with it. What do you do?

A slow dependency is worse than a dead one: requests pile up, threads and connections run out, and your service fails even for callers who never needed that dependency.

1. Timeouts on every outbound call, set from the dependency's latency percentiles, not library defaults.
2. Circuit breaker: closed counts failures; past an error threshold it goes open and fails fast for a cool-down; then half-open lets a few trial calls through, closing on success and reopening on failure.
3. Bulkheads: a separate bounded pool or concurrency limit per dependency, so one bad dependency only exhausts its own share.
4. Fallbacks: cached or degraded responses ("recommendations unavailable") instead of failing the page.

Follow-up on thresholds: require a minimum request volume as well as an error rate, so three failures out of three requests at 3 a.m. don't trip it - and alert when a breaker opens.

47. A service had a 5-minute outage, came back, and immediately fell over again - repeatedly. Why?

A retry storm, possibly a metastable failure: overload that sustains itself after the trigger is gone (Bronson et al. named the pattern in 2021). On recovery the service faces normal load plus every queued retry plus a cold cache. Latency climbs, clients time out and retry again, and the service burns capacity on requests whose callers have already given up.

Fixes:

• Load shedding: reject excess work cheaply and early (503 with Retry-After), lowest-priority traffic first.
• Backpressure: bounded queues everywhere, pushing back on callers instead of buffering. By Little's Law (L = λW), an unbounded queue just converts overload into latency.
• Client discipline: jittered backoff, retry budgets, deadlines (question 45).
• Recovery: ramp traffic back gradually, and drop or deprioritize stale queued requests so you serve ones still wanted.

48. Users report that after they post a comment and refresh, the comment is sometimes missing. Diagnose it.

Hypothesis: writes go to the primary and reads come from asynchronously replicated replicas, so the refresh sometimes lands on a replica that hasn't applied the write - a read-your-writes violation. "Sometimes" fits, because it depends on which replica serves the read and on current lag. Confirm by correlating reports with replica lag metrics.

Fixes, cheapest first:

• Route a user's reads to the primary for a short window after they write (longer than p99 lag).
• Return a version token with the write and only read from replicas caught up to it.
• Render the user's own comment optimistically client-side - but say that only fixes the author's view.

Then ask the product question: does anyone else need to see it immediately? Usually not, so read-your-writes, not global linearizability, is the right target.

49. A nightly job is protected by a distributed lock, but sometimes runs twice and corrupts data. Why, and how do you fix it?

Likely causes, in the order I'd check:

1. The lease expired mid-run: the job outlived its TTL, or paused (GC, slow disk, VM migration) and missed a renewal, and a second worker took the lock.
2. The lock isn't safe: a single Redis primary with asynchronous replication can lose a grant on failover.
3. The scheduler fired twice: two scheduler instances, or a retry after a timeout.

Fixes:

• Fencing tokens (question 29), checked by the data store - the one fix that holds under arbitrary pauses.
• Renew the lease from a background heartbeat and re-check ownership before each batch.
• Make the job idempotent: checkpointed batches keyed by run ID, so a duplicate run is a no-op.
• Use etcd or ZooKeeper if correctness depends on the lock.

The key sentence: a lock alone is fine for efficiency; for correctness, the resource must enforce it.

50. Your 3-node etcd cluster spans two availability zones. One AZ fails and writes stop. Explain, and redesign it.

With nodes split 2 + 1, losing the two-node AZ leaves one node - not a majority of three. It correctly refuses writes: it can't tell "the other AZ is gone" from "I'm the one partitioned," and accepting writes would risk split brain. Consensus is working as designed.

Redesign:

• Three AZs, one node each. Losing any AZ leaves 2 of 3. The standard answer.
• Five nodes across three AZs (2 + 2 + 1) to survive an AZ loss plus another node failure.
• With only two sites, add a witness member in a third location. Adding nodes within two sites doesn't help: one site always holds the majority, or they tie.

Follow-up: cross-AZ latency is added to every commit (PACELC's "else latency"). Spanning regions makes that tens of milliseconds per write, which is why consensus groups usually stay within a region with asynchronous replication between regions.

How to practice for a distributed systems interview

• Draw the failure, not just the happy path. For each arrow in a design, ask "what if this message is delayed 30 seconds, delivered twice, or lost after the work was done?"
• Read two papers properly. Raft and Dynamo together cover consensus, replication, quorums, vector clocks, hinted handoff and anti-entropy.
• Be precise with words. Say "linearizable," "serializable" or "read-your-writes" instead of "consistent." Many follow-ups exist to test exactly that.
• Practise scenarios out loud. Symptom, ranked hypotheses, how you'd confirm, then the fix and its cost - about ten minutes each.
• Run a real cluster. Start a 3-node etcd cluster locally, kill the leader, watch the election, and try writing to an isolated minority node.
• Tie every mechanism to its cost. Quorums cost latency, fencing needs storage support, CRDTs can't enforce invariants, sagas give up isolation. The trade-off is the answer.

Keep going

• Drill it under pressure: the system design and databases question banks cover consistency, replication and partitioning as timed multiple-choice, and the free daily challenge gives you one question a day with no account.
• Keep the references open: the distributed systems patterns, system design tradeoffs, concurrency essentials and Kafka and messaging cheat sheets.
• Go deeper on the hard parts: the consensus and distributed transactions walkthroughs, plus consistent hashing explained simply.
• Round out the loop: Top 30 system design interview questions, Top 50 microservices interview questions, Top 50 Kafka interview questions and Top 50 SRE interview questions. For a structured path, the backend junior-to-senior track sequences this material with the rest of a backend loop.
• Practise the follow-ups: the chat-based AI mock interview pushes on "and what if the leader pauses for ten seconds?" the way a real distributed systems round does.
• Related guides: Top 50 DynamoDB Interview Questions 2026, Top 50 AWS Lambda and Serverless Interview Questions 2026, Top 50 Spring Boot Interview Questions 2026.

#distributedsystems #systemdesign #interviews #backend #sre #consensus #career