What's the scale - thousands of services, hundreds of thousands? Million-host fleet or smaller?

10M metric data points/sec ingest. 100 TB/day logs. 1B traces/day, 10B spans/day. 1000 unique services. 10K engineers as users. 100K alerting rules.

gitGood.dev

Distributed Systems

Design an Observability Platform (Metrics, Logs, Traces)

Hard Premium

Time-series DBs (Prometheus, M3, VictoriaMetrics), trace sampling, exemplars, OpenTelemetry, alerting, and the cardinality explosion that turns a $10K/month platform into a $1M/month outage.

The problem

Design the observability platform for a 10,000-engineer company. Hundreds of services, billions of metric data points per day, terabytes of logs, and traces spanning 50+ hops per user request. Engineers must diagnose incidents in minutes, not hours. The platform itself must cost less than 5% of total infra spend - which is a hard constraint at this scale, not a nice-to-have.

This is the question that separates "I configured Datadog" from "I have run an observability fleet". Strong candidates separate the three signals (metrics, logs, traces), explain cardinality math, design the sampling story for traces, and own the cost model. Excellent candidates discuss exemplars (the bridge that links a metric anomaly to the trace that caused it) and explain why pure tail-based sampling has scaled poorly historically.

Clarifying questions

Asking these before diving into a solution is the difference between a "hire" and a "no signal" rating. Pick the questions whose answers would change your design.

→What's the scale - thousands of services, hundreds of thousands? Million-host fleet or smaller?
→Are we building all three signals (metrics, logs, traces) or starting with one?
→Retention - 7 days, 30 days, 1 year per signal? Compliance requirements?
→Engineers expected to write their own queries, or are we curating dashboards?
→Multi-tenant (each team isolated) or shared?
→Self-hosted on-prem, managed cloud (Datadog/New Relic), or open source self-managed (Grafana stack)?
→Real-time alerting (sub-minute) or eventual (5-15 min lag acceptable)?
→Does the SRE org own the platform, or is it a paid-for product to internal teams?

Requirements

Functional requirements

·Ingest metrics (numeric time series, labels), logs (structured + unstructured), and traces (request spans across services)
·Query each signal type with appropriate languages (PromQL, LogQL, traceQL)
·Cross-link signals: a metric anomaly should jump to the relevant logs and traces
·Alerting on metric expressions with deduplication and routing
·Dashboards: composable, version-controlled, sharable
·Per-team multi-tenancy with quotas, isolation, and cost attribution
·Sampling strategies for traces (head-based, tail-based, rule-based)

Non-functional requirements

Scale: 10M metric data points/sec ingest. 100 TB/day logs. 1B traces/day, 10B spans/day. 1000 unique services. 10K engineers as users. 100K alerting rules.
Latency: Metric ingest → query visibility: <30s. Log ingest → search: <1min. Trace assembly → query: <2min. Alert firing latency: <60s for critical. Dashboard query p99: <5s.
Availability: 99.9% for ingest (data loss is unacceptable during incidents - that's exactly when you need it most). 99% for query (degraded query is recoverable; lost ingest is not).
Consistency: Eventual. A metric written now might be queryable in 5s. A log might be searchable in 30s. Traces require all spans before assembly - up to minute-scale lag is acceptable.

The full walkthrough covers

Architecture components

·OpenTelemetry collector / agent
·Metrics ingest gateway
·Time-series database (Prometheus / M3 / VictoriaMetrics / Mimir)
·Logs ingest pipeline
·Trace ingest + assembler
·Storage tiering
·Query engine
·Alerting engine
·Dashboard layer (Grafana)
·Self-monitoring stack

Deep dives

Cardinality explosion: the failure mode that kills naive deployments
Trace sampling: head, tail, and rule-based
Time-series databases: Prometheus, M3, VictoriaMetrics, Mimir, Cortex
Logs at scale: indexed vs grep-on-object-storage
OpenTelemetry: why it matters and the migration path
Alerting: dedup, routing, and the on-call's pager

What's in the full walkthrough

Capacity estimation · architecture with all 10 components explained · 6 deep dives · trade-off analysis · 8 common follow-up questions

Subscribe to Continue

Get full access to gitGood.dev with a subscription.

Start Your 7-Day Trial

Try everything free for 7 days. Cancel anytime.

2,000+ practice questions
AI mock interviews (3/day)
Coding challenges with real execution
Verified skill badges
146 certification guides, 26 with practice exams

View Plans

7-day trial, then $8/mo - or $64/yr ($5.33/mo, save 33%). Cancel anytime.

Companies that test this topic

Practice in interview format

Reading is the floor. The interview signal is in walking through this live with someone probing follow-ups. Use the AI mock interview to practice talking through requirements, architecture, and trade-offs out loud.

Start an AI mock interview →

Distributed Systems

Design an Observability Platform (Metrics, Logs, Traces)

Hard Premium

Time-series DBs (Prometheus, M3, VictoriaMetrics), trace sampling, exemplars, OpenTelemetry, alerting, and the cardinality explosion that turns a $10K/month platform into a $1M/month outage.

The problem

Clarifying questions

Asking these before diving into a solution is the difference between a "hire" and a "no signal" rating. Pick the questions whose answers would change your design.

→What's the scale - thousands of services, hundreds of thousands? Million-host fleet or smaller?
→Are we building all three signals (metrics, logs, traces) or starting with one?
→Retention - 7 days, 30 days, 1 year per signal? Compliance requirements?
→Engineers expected to write their own queries, or are we curating dashboards?
→Multi-tenant (each team isolated) or shared?
→Self-hosted on-prem, managed cloud (Datadog/New Relic), or open source self-managed (Grafana stack)?
→Real-time alerting (sub-minute) or eventual (5-15 min lag acceptable)?
→Does the SRE org own the platform, or is it a paid-for product to internal teams?

Requirements

Functional requirements

·Ingest metrics (numeric time series, labels), logs (structured + unstructured), and traces (request spans across services)
·Query each signal type with appropriate languages (PromQL, LogQL, traceQL)
·Cross-link signals: a metric anomaly should jump to the relevant logs and traces
·Alerting on metric expressions with deduplication and routing
·Dashboards: composable, version-controlled, sharable
·Per-team multi-tenancy with quotas, isolation, and cost attribution
·Sampling strategies for traces (head-based, tail-based, rule-based)

Non-functional requirements

Scale: 10M metric data points/sec ingest. 100 TB/day logs. 1B traces/day, 10B spans/day. 1000 unique services. 10K engineers as users. 100K alerting rules.
Latency: Metric ingest → query visibility: <30s. Log ingest → search: <1min. Trace assembly → query: <2min. Alert firing latency: <60s for critical. Dashboard query p99: <5s.
Availability: 99.9% for ingest (data loss is unacceptable during incidents - that's exactly when you need it most). 99% for query (degraded query is recoverable; lost ingest is not).
Consistency: Eventual. A metric written now might be queryable in 5s. A log might be searchable in 30s. Traces require all spans before assembly - up to minute-scale lag is acceptable.

The full walkthrough covers

Architecture components

·OpenTelemetry collector / agent
·Metrics ingest gateway
·Time-series database (Prometheus / M3 / VictoriaMetrics / Mimir)
·Logs ingest pipeline
·Trace ingest + assembler
·Storage tiering
·Query engine
·Alerting engine
·Dashboard layer (Grafana)
·Self-monitoring stack

Deep dives

Cardinality explosion: the failure mode that kills naive deployments
Trace sampling: head, tail, and rule-based
Time-series databases: Prometheus, M3, VictoriaMetrics, Mimir, Cortex
Logs at scale: indexed vs grep-on-object-storage
OpenTelemetry: why it matters and the migration path
Alerting: dedup, routing, and the on-call's pager

What's in the full walkthrough

Capacity estimation · architecture with all 10 components explained · 6 deep dives · trade-off analysis · 8 common follow-up questions

Subscribe to Continue

Get full access to gitGood.dev with a subscription.

Start Your 7-Day Trial

Try everything free for 7 days. Cancel anytime.

2,000+ practice questions
AI mock interviews (3/day)
Coding challenges with real execution
Verified skill badges
146 certification guides, 26 with practice exams

View Plans

7-day trial, then $8/mo - or $64/yr ($5.33/mo, save 33%). Cancel anytime.

Companies that test this topic

Practice in interview format

Start an AI mock interview →

Design an Observability Platform (Metrics, Logs, Traces)

The problem

Clarifying questions

Requirements

Functional requirements

Non-functional requirements

The full walkthrough covers

Architecture components

Deep dives

What's in the full walkthrough

Subscribe to Continue

Start Your 7-Day Trial

Related system design topics

Analytics Pipeline

Consensus

Companies that test this topic

Practice in interview format

Design an Observability Platform (Metrics, Logs, Traces)

The problem

Clarifying questions

Requirements

Functional requirements

Non-functional requirements

The full walkthrough covers

Architecture components

Deep dives

What's in the full walkthrough

Subscribe to Continue

Start Your 7-Day Trial

Related system design topics

Analytics Pipeline

Consensus

Companies that test this topic

Practice in interview format