Most candidates design the happy path. They draw the boxes, connect the arrows, and explain how a request flows from client to database and back. Then the interviewer asks "what happens when that database goes down?" and the whole thing stalls.
That question isn't a trap. It's the actual point of the interview. Real systems fail constantly. Disks fill up, networks drop packets, a deploy goes sideways at 2am. The interviewer wants to know if you've thought about that before, because in production you don't get to assume things work.
Here's how to show them you have.
Bring up failure before they ask
The strongest signal you can send is raising failure yourself. When you finish a component, don't wait for the prompt. Say "this service is a single point of failure right now, so let me talk about what happens when it dies."
That one sentence changes how the interviewer reads you. You've gone from someone who draws diagrams to someone who's operated systems. You don't need a perfect answer. You need to show the instinct.
A simple habit: after you place any box on the board, ask yourself out loud "what breaks if this disappears?" Do that two or three times during the session and you'll cover more ground than most candidates.
Know the failure modes worth naming
You don't have to memorize a textbook. A handful of failure types cover most of what comes up.
- A single instance crashes. The fix is usually more than one instance behind a load balancer, plus health checks that pull the dead one out.
- A whole zone or region goes down. Now you're talking replication across zones, and whether you can tolerate the latency of a failover.
- A dependency gets slow instead of dying. This one trips people up. A slow service is often worse than a dead one because it ties up your threads and connections. Timeouts and circuit breakers matter here.
- The network partitions. Two halves of your system can't talk but both are alive. You have to choose what to give up, consistency or availability, and say why.
- Traffic spikes past what you planned for. Rate limiting, queues, and load shedding keep the system alive instead of letting it fall over completely.
Pick the ones that fit the system you're designing and go deep on a couple rather than listing all five.
Talk about what happens to data
When something fails mid-write, what state are you left in? This is where you separate a real answer from hand-waving.
If a payment service charges the card but crashes before recording the order, you've taken money and delivered nothing. So you talk about idempotency keys, so a retry doesn't double-charge. You talk about whether you need a transaction, or whether you can use an outbox pattern to make the write and the message atomic enough.
You don't need the perfect distributed-systems solution. You need to show you know the data can end up in a bad state and you have a plan for detecting and fixing it.
Degrade instead of dying
A system that returns slightly worse results beats one that returns an error page. Interviewers love it when you reach for this on your own.
If the recommendation service is down, serve a generic list instead of failing the whole page. If the live inventory count times out, show a cached number and a note that it might be stale. Name the tradeoff out loud. You're choosing a slightly wrong answer over no answer, and that's usually the right call for a user-facing path.
Say how you'd know it broke
Designing for failure isn't only about recovery. It's about noticing. A candidate who adds "and I'd put a metric on the queue depth and alert if it backs up" sounds like someone who's been paged before.
You don't need a full observability stack on the whiteboard. Just point at the riskiest spots and say what you'd watch: error rates, latency at the tail, depth of any queue, and whether retries are climbing. That's enough to show you'd catch the problem before a customer does.
A short script you can reuse
When the failure question comes, walk it in this order:
- Name what failed and how. "The cache node dies."
- Say the immediate impact. "Every read now hits the database directly, and that's a 5x load jump."
- Give the mitigation. "So I'd run a small cluster, and add request coalescing so a cache miss doesn't trigger a thundering herd."
- Admit the cost. "That adds complexity and a bit of latency, but it's worth it to survive a node loss."
Four steps, and you've shown the whole loop: detection, impact, fix, tradeoff.
The mindset that comes through
What interviewers are really checking is whether you treat failure as normal. Junior answers assume things work and patch around exceptions. Stronger answers assume things break and design so the break is survivable.
You can practice this. Take any system you've built or studied, point at each piece, and ask what happens when it fails. Mock interviews help here because saying it out loud is a different skill than thinking it. Do enough reps and the failure question stops feeling like an ambush and starts feeling like the part of the interview where you get to show what you actually know.