We use analytics and advertising cookies to understand how the site is used and whether our ads on Facebook and Instagram work. They are set only if you accept. See our Privacy Policy for details.
Politeness, deduplication, freshness, and the URL frontier. The classic crawl-the-internet question that surfaces deep distributed systems judgment.
Design a web crawler: a distributed system that fetches web pages, extracts links, follows them, stores content for downstream indexing, and returns to refresh pages over time. The output is a continuously updated corpus of the public web.
This is a foundational distributed systems question. Strong candidates demonstrate fluency in the URL frontier (the queue of work), politeness (don't hammer one host), deduplication (URL canonicalization, content hashing), and the freshness / coverage trade-off.
Asking these before diving into a solution is the difference between a "hire" and a "no signal" rating. Pick the questions whose answers would change your design.
Capacity estimation · architecture with all 8 components explained · 6 deep dives · trade-off analysis · 8 common follow-up questions
Unlock it with a subscription.
Try everything free for 7 days. Cancel anytime.
7-day trial, then $8/mo - or $48/yr ($4/mo, save 50%). Cancel anytime.
Consistent hashing, eviction, replication, and what really happens when a single hot key takes down the cluster.
Five algorithms, three sharding strategies, one fail-open vs fail-closed decision. The bounded design that surfaces in every backend interview loop.
Reading is the floor. The interview signal is in walking through this live with someone probing follow-ups. Use the AI mock interview to practice talking through requirements, architecture, and trade-offs out loud.
Mock interview: design Web Crawler →