gitGood.dev

Distributed Systems

Design a Web Crawler

Hard Premium

Politeness, deduplication, freshness, and the URL frontier. The classic crawl-the-internet question that surfaces deep distributed systems judgment.

The problem

Design a web crawler: a distributed system that fetches web pages, extracts links, follows them, stores content for downstream indexing, and returns to refresh pages over time. The output is a continuously updated corpus of the public web.

This is a foundational distributed systems question. Strong candidates demonstrate fluency in the URL frontier (the queue of work), politeness (don't hammer one host), deduplication (URL canonicalization, content hashing), and the freshness / coverage trade-off.

Clarifying questions

Asking these before diving into a solution is the difference between a "hire" and a "no signal" rating. Pick the questions whose answers would change your design.

→What's the scope - the entire public web, a specific domain set, or a vertical (news, ecommerce)?
→What's the target corpus size - 1B pages, 100B pages, or a small set?
→How fresh does content need to be - news refresh in minutes, evergreen content in weeks?
→What's downstream - search indexing, ML training data, archive, or alerting?
→Do we need to render JavaScript (full browser) or only static HTML?
→Are we crawling adversarial sites - cloaking, anti-bot defenses, login walls?
→What's the bandwidth budget - we control how much of someone's site we hit per minute?
→Are we respecting robots.txt strictly, or are there business reasons to override?

Requirements

Functional requirements

·Fetch HTML pages from a URL frontier
·Extract outbound links and add them to the frontier
·Deduplicate URLs (canonicalization) and content (fingerprinting)
·Respect robots.txt and crawl-delay directives
·Politeness: rate-limit requests per host
·Schedule re-crawls based on observed change frequency
·Store fetched content for downstream indexing

Non-functional requirements

Scale: 100B pages in corpus. 1B fetches/day (re-crawl + new). 100M unique hosts. 200KB average page size. 5-year retention with tiered storage.
Latency: Per-fetch latency depends on the target server (we don't control it). Internal pipeline latency from frontier → fetch → store should add < 100ms.
Availability: 99.9% on the crawl pipeline. A 1-hour outage means missing 40M pages worth of refresh - recoverable but visible. Storage availability: 99.99%.
Consistency: Eventually consistent. The crawler is fundamentally an eventually-consistent view of the web. Strong consistency would mean stop-the-world snapshots, which doesn't fit the workload.

The full walkthrough covers

Architecture components

·URL frontier
·DNS resolver
·Fetcher pool
·Parser
·Dedupe service
·Content store
·Robots.txt cache
·Scheduler

Deep dives

URL frontier: the heart of the crawler
Politeness and robots.txt
Deduplication: URLs and content
Freshness: when to re-crawl
Crawl traps and adversarial sites
Storage layout and downstream consumers

What's in the full walkthrough

Capacity estimation · architecture with all 8 components explained · 6 deep dives · trade-off analysis · 8 common follow-up questions

Subscribe to Continue

Get full access to gitGood.dev with a subscription.

Start Your 10-Day Trial

Try everything free for 10 days. Cancel anytime.

1,950+ practice questions
AI mock interviews (3/day)
Coding challenges with real execution
Verified skill badges
146 certification guides, 26 with practice exams

View Plans

10-day trial, then $5/mo - or $40/yr ($3.33/mo, save 33%). Cancel anytime.

Companies that test this topic

Google

L4 Software Engineer

Practice in interview format

Reading is the floor. The interview signal is in walking through this live with someone probing follow-ups. Use the AI mock interview to practice talking through requirements, architecture, and trade-offs out loud.

Start an AI mock interview →

Distributed Systems

Design a Web Crawler

Hard Premium

Politeness, deduplication, freshness, and the URL frontier. The classic crawl-the-internet question that surfaces deep distributed systems judgment.

The problem

Clarifying questions

Asking these before diving into a solution is the difference between a "hire" and a "no signal" rating. Pick the questions whose answers would change your design.

→What's the scope - the entire public web, a specific domain set, or a vertical (news, ecommerce)?
→What's the target corpus size - 1B pages, 100B pages, or a small set?
→How fresh does content need to be - news refresh in minutes, evergreen content in weeks?
→What's downstream - search indexing, ML training data, archive, or alerting?
→Do we need to render JavaScript (full browser) or only static HTML?
→Are we crawling adversarial sites - cloaking, anti-bot defenses, login walls?
→What's the bandwidth budget - we control how much of someone's site we hit per minute?
→Are we respecting robots.txt strictly, or are there business reasons to override?

Requirements

Functional requirements

·Fetch HTML pages from a URL frontier
·Extract outbound links and add them to the frontier
·Deduplicate URLs (canonicalization) and content (fingerprinting)
·Respect robots.txt and crawl-delay directives
·Politeness: rate-limit requests per host
·Schedule re-crawls based on observed change frequency
·Store fetched content for downstream indexing

Non-functional requirements

Scale: 100B pages in corpus. 1B fetches/day (re-crawl + new). 100M unique hosts. 200KB average page size. 5-year retention with tiered storage.
Latency: Per-fetch latency depends on the target server (we don't control it). Internal pipeline latency from frontier → fetch → store should add < 100ms.
Availability: 99.9% on the crawl pipeline. A 1-hour outage means missing 40M pages worth of refresh - recoverable but visible. Storage availability: 99.99%.
Consistency: Eventually consistent. The crawler is fundamentally an eventually-consistent view of the web. Strong consistency would mean stop-the-world snapshots, which doesn't fit the workload.

The full walkthrough covers

Architecture components

·URL frontier
·DNS resolver
·Fetcher pool
·Parser
·Dedupe service
·Content store
·Robots.txt cache
·Scheduler

Deep dives

URL frontier: the heart of the crawler
Politeness and robots.txt
Deduplication: URLs and content
Freshness: when to re-crawl
Crawl traps and adversarial sites
Storage layout and downstream consumers

What's in the full walkthrough

Capacity estimation · architecture with all 8 components explained · 6 deep dives · trade-off analysis · 8 common follow-up questions

Subscribe to Continue

Get full access to gitGood.dev with a subscription.

Start Your 10-Day Trial

Try everything free for 10 days. Cancel anytime.

1,950+ practice questions
AI mock interviews (3/day)
Coding challenges with real execution
Verified skill badges
146 certification guides, 26 with practice exams

View Plans

10-day trial, then $5/mo - or $40/yr ($3.33/mo, save 33%). Cancel anytime.

Companies that test this topic

Google

L4 Software Engineer

Practice in interview format

Start an AI mock interview →

Design a Web Crawler

The problem

Clarifying questions

Requirements

Functional requirements

Non-functional requirements

The full walkthrough covers

Architecture components

Deep dives

What's in the full walkthrough

Subscribe to Continue

Start Your 10-Day Trial

Related system design topics

Distributed Cache

Rate Limiter

Companies that test this topic

Practice in interview format

Design a Web Crawler

The problem

Clarifying questions

Requirements

Functional requirements

Non-functional requirements

The full walkthrough covers

Architecture components

Deep dives

What's in the full walkthrough

Subscribe to Continue

Start Your 10-Day Trial

Related system design topics

Distributed Cache

Rate Limiter

Companies that test this topic

Practice in interview format