What's the event volume - 10K, 1M, or 100M events/sec at peak?

10M events/sec at peak. 1 PB ingested/day. 50 PB warehouse. 5,000 active analysts running queries. 200 production dashboards refreshed every 1-60 seconds.

What latency tiers - real-time (seconds), near-real-time (minutes), batch (hours)?

Real-time path p99 < 10s end-to-end (event emission → dashboard). Batch path p95 < 60 minutes (event → warehouse table). Analyst queries p95 < 30s on warm partitions.

gitGood.dev

Streaming

Design an Analytics Pipeline (Kafka / Spark / Warehouse)

Hard Premium

Batch vs streaming, lambda vs kappa, the warehouse-vs-lakehouse decision, and dimension modeling that survives schema drift.

The problem

Design a pipeline that ingests every user-facing event the product produces, makes the freshest data available for real-time dashboards within seconds, makes the cleanest data available for analyst queries within hours, and never loses an event. Schemas evolve weekly; new event types appear; old ones get repurposed.

This is the systems analog of "design Netflix data infra." The interview signal is whether the candidate can defend lambda vs kappa, separate ingestion from transformation, and explain why the warehouse and the streaming store are not the same database. Strong candidates draw the timeline of a single event from emission to dashboard and identify where it can break.

Clarifying questions

Asking these before diving into a solution is the difference between a "hire" and a "no signal" rating. Pick the questions whose answers would change your design.

→What's the event volume - 10K, 1M, or 100M events/sec at peak?
→What latency tiers - real-time (seconds), near-real-time (minutes), batch (hours)?
→What downstream consumers - dashboards, ML training, ad-hoc SQL, business reporting, third-party export?
→Schema strictness - schema-on-write (enforced at ingest) or schema-on-read (raw payloads, parsed at query time)?
→Privacy requirements - PII redaction, GDPR right-to-delete propagating through the pipeline?
→Multi-tenant - one platform serving many product teams with isolation?
→Cost ceiling - is the pipeline expected to run at $1M/year or $50M/year of cloud spend?
→Reprocessing - how often do we need to rerun a transform on historical data after a bug fix?

Requirements

Functional requirements

·Ingest events from web, mobile, backend services, third-party webhooks
·Real-time path: events queryable on a streaming dashboard within ~10 seconds
·Batch path: events available in the warehouse for SQL within ~1 hour
·Schema registry with versioning and backward-compatibility rules
·Reprocess historical data on demand (replay last 30 days through a fixed transform)
·Per-event lineage - track which dataset / dashboard a given source event flows into
·GDPR delete - propagate user_id deletes through warm and cold storage within SLA

Non-functional requirements

Scale: 10M events/sec at peak. 1 PB ingested/day. 50 PB warehouse. 5,000 active analysts running queries. 200 production dashboards refreshed every 1-60 seconds.
Latency: Real-time path p99 < 10s end-to-end (event emission → dashboard). Batch path p95 < 60 minutes (event → warehouse table). Analyst queries p95 < 30s on warm partitions.
Availability: 99.9% for ingestion (events must not be lost). 99% for batch (a delayed batch is recoverable). Real-time dashboards can degrade to last-known-good during outages.
Consistency: At-least-once ingestion with idempotency at the consumer (event_id dedup). Streaming aggregates are eventually consistent and may be revised when late events arrive. Warehouse tables are strongly consistent within a partition once batch finishes.

The full walkthrough covers

Architecture components

·Ingest API
·Kafka (event log)
·Schema registry
·Stream processor (Flink / Spark Streaming)
·Serving store (Pinot / Druid / ClickHouse)
·Object storage (S3)
·Lakehouse table format (Iceberg / Delta / Hudi)
·Batch processor (Spark / dbt)
·Warehouse query engine (Snowflake / BigQuery / Trino)
·Semantic layer / metric store
·Lineage / observability

Deep dives

Lambda vs kappa: pick one and own the trade-off
Warehouse vs lakehouse vs OLAP store
Dimension modeling and the schema-evolution problem
Reprocessing and backfill
Data quality, freshness, and lineage
Cost: where the money goes and how to cut it

What's in the full walkthrough

Capacity estimation · architecture with all 11 components explained · 6 deep dives · trade-off analysis · 8 common follow-up questions

Subscribe to Continue

Get full access to gitGood.dev with a subscription.

Start Your 10-Day Trial

Try everything free for 10 days. Cancel anytime.

1,950+ practice questions
AI mock interviews (3/day)
Coding challenges with real execution
Verified skill badges
146 certification guides, 26 with practice exams

View Plans

10-day trial, then $5/mo - or $40/yr ($3.33/mo, save 33%). Cancel anytime.

Companies that test this topic

Practice in interview format

Reading is the floor. The interview signal is in walking through this live with someone probing follow-ups. Use the AI mock interview to practice talking through requirements, architecture, and trade-offs out loud.

Start an AI mock interview →

Streaming

Design an Analytics Pipeline (Kafka / Spark / Warehouse)

Hard Premium

Batch vs streaming, lambda vs kappa, the warehouse-vs-lakehouse decision, and dimension modeling that survives schema drift.

The problem

Clarifying questions

Asking these before diving into a solution is the difference between a "hire" and a "no signal" rating. Pick the questions whose answers would change your design.

→What's the event volume - 10K, 1M, or 100M events/sec at peak?
→What latency tiers - real-time (seconds), near-real-time (minutes), batch (hours)?
→What downstream consumers - dashboards, ML training, ad-hoc SQL, business reporting, third-party export?
→Schema strictness - schema-on-write (enforced at ingest) or schema-on-read (raw payloads, parsed at query time)?
→Privacy requirements - PII redaction, GDPR right-to-delete propagating through the pipeline?
→Multi-tenant - one platform serving many product teams with isolation?
→Cost ceiling - is the pipeline expected to run at $1M/year or $50M/year of cloud spend?
→Reprocessing - how often do we need to rerun a transform on historical data after a bug fix?

Requirements

Functional requirements

·Ingest events from web, mobile, backend services, third-party webhooks
·Real-time path: events queryable on a streaming dashboard within ~10 seconds
·Batch path: events available in the warehouse for SQL within ~1 hour
·Schema registry with versioning and backward-compatibility rules
·Reprocess historical data on demand (replay last 30 days through a fixed transform)
·Per-event lineage - track which dataset / dashboard a given source event flows into
·GDPR delete - propagate user_id deletes through warm and cold storage within SLA

Non-functional requirements

Scale: 10M events/sec at peak. 1 PB ingested/day. 50 PB warehouse. 5,000 active analysts running queries. 200 production dashboards refreshed every 1-60 seconds.
Latency: Real-time path p99 < 10s end-to-end (event emission → dashboard). Batch path p95 < 60 minutes (event → warehouse table). Analyst queries p95 < 30s on warm partitions.
Availability: 99.9% for ingestion (events must not be lost). 99% for batch (a delayed batch is recoverable). Real-time dashboards can degrade to last-known-good during outages.
Consistency: At-least-once ingestion with idempotency at the consumer (event_id dedup). Streaming aggregates are eventually consistent and may be revised when late events arrive. Warehouse tables are strongly consistent within a partition once batch finishes.

The full walkthrough covers

Architecture components

·Ingest API
·Kafka (event log)
·Schema registry
·Stream processor (Flink / Spark Streaming)
·Serving store (Pinot / Druid / ClickHouse)
·Object storage (S3)
·Lakehouse table format (Iceberg / Delta / Hudi)
·Batch processor (Spark / dbt)
·Warehouse query engine (Snowflake / BigQuery / Trino)
·Semantic layer / metric store
·Lineage / observability

Deep dives

Lambda vs kappa: pick one and own the trade-off
Warehouse vs lakehouse vs OLAP store
Dimension modeling and the schema-evolution problem
Reprocessing and backfill
Data quality, freshness, and lineage
Cost: where the money goes and how to cut it

What's in the full walkthrough

Capacity estimation · architecture with all 11 components explained · 6 deep dives · trade-off analysis · 8 common follow-up questions

Subscribe to Continue

Get full access to gitGood.dev with a subscription.

Start Your 10-Day Trial

Try everything free for 10 days. Cancel anytime.

1,950+ practice questions
AI mock interviews (3/day)
Coding challenges with real execution
Verified skill badges
146 certification guides, 26 with practice exams

View Plans

10-day trial, then $5/mo - or $40/yr ($3.33/mo, save 33%). Cancel anytime.

Companies that test this topic

Practice in interview format

Start an AI mock interview →

Design an Analytics Pipeline (Kafka / Spark / Warehouse)

The problem

Clarifying questions

Requirements

Functional requirements

Non-functional requirements

The full walkthrough covers

Architecture components

Deep dives

What's in the full walkthrough

Subscribe to Continue

Start Your 10-Day Trial

Related system design topics

Notifications

Search + Autocomplete

Companies that test this topic

Practice in interview format

Design an Analytics Pipeline (Kafka / Spark / Warehouse)

The problem

Clarifying questions

Requirements

Functional requirements

Non-functional requirements

The full walkthrough covers

Architecture components

Deep dives

What's in the full walkthrough

Subscribe to Continue

Start Your 10-Day Trial

Related system design topics

Notifications

Search + Autocomplete

Companies that test this topic

Practice in interview format