Design an Observability Platform (Metrics, Logs, Traces)

Cardinality is the count of unique time series. It dominates cost and is the single most common cause of platform outages.

The math
A metric is identified by its name + label set. Each unique combination of labels is a separate time series.

http_requests_total{method, status, endpoint, user_id, region}

If method has 5 values, status has 10, endpoint has 100, user_id has 10M, and region has 5:
5 × 10 × 100 × 10M × 5 = 250 billion time series.

Even if each series is small (~1KB resident memory), 250B series = 250 TB. The TSDB OOMs.

The most common offender: unbounded labels

• user_id, request_id, session_id, transaction_id - high-cardinality identifiers attached to metrics.
• URLs with path parameters (/users/123, /users/124, ...) - each becomes a unique label value.
• Stack traces in error metrics.
• Email addresses, IP addresses in development metrics.

These almost always sneak in via well-meaning instrumentation: "let's tag this metric with user_id so we can break it down by user". The result is a platform-killing label.

Detection

• Track time-series-count per metric. Alert when a metric exceeds 100K series.
• Track time-series-growth-rate. A metric whose series count doubled in an hour is a fire drill.
• Per-tenant series count + budget enforcement.

Mitigation

1. Cap at ingest: reject writes that exceed per-metric cardinality budgets. Painful but effective.
2. Drop-the-label: when ingest detects an exploding label, drop that label automatically. Loses the dimension but preserves the metric.
3. Aggregate: instead of per-user counters, emit per-cohort counters (10 user-segment buckets, not 10M user_id values).
4. Move to logs/traces: if the dimension matters per-user, store it as logs or traces (which scale by volume, not by series count) and use exemplars to link.

Educational layer
The platform must teach engineers cardinality discipline. Pre-deploy linting on metric names + labels. Onboarding docs. Per-team office hours. Detected explosions get a Slack ping with the offending metric and the suggested fix.

The "we'll fix it later" trap
Once a high-cardinality metric is in production, removing the label feels like a breaking change ("but my dashboard uses it"). It accumulates. The platform team's job is to enforce limits at ingest, not at policy-discussion time.

The right mental model
Metrics are aggregations. If you find yourself wanting per-event detail, you want logs or traces, not metrics. Treat metrics as "the questions I ask without specifying the entity"; logs/traces as "the questions I ask about specific entities".

Tracing every request at full detail is unaffordable at scale. Sampling is mandatory; the strategy determines which incidents you can debug.

Head-based sampling
Decision is made at the start of the trace, propagated to all child spans. Most commonly: keep N% of traces uniformly.

Pros: cheap (drop spans at the originating service, never collected). Decision is consistent across a trace (no half-traces).
Cons: you keep N% of normal traces and N% of error traces. Errors are rare; sampling them at 1% means most errors have no trace.

Use when: you mostly care about typical request shape, not rare incidents.

Rule-based head sampling
Sample at higher rates for specific cases: 100% of traces with a debug header, 100% from a specific test account, 50% from a specific endpoint.

Pros: targeted depth without exploding cost.
Cons: requires per-rule maintenance; rules drift.

Use when: you have known-important request types worth full visibility.

Tail-based sampling
Wait until the trace completes; decide based on outcome. Keep 100% of error traces, 100% of traces > p99 latency, X% of normal.

Pros: every error has a trace. Slow traces get analyzed. Headroom isn't wasted on uneventful requests.
Cons: must buffer all spans for ~30s before sampling decision. Memory and CPU cost. Hard to do across services without a centralized assembler.

Use when: you have the infrastructure to buffer + decide. Production-grade observability tools (Tempo, Honeycomb, Lightstep, Jaeger with collector) all support tail sampling.

Hybrid: head + tail
Head-sample at 100% (or close to it) into a temporary buffer. Tail-sample to decide what to persist long-term. The "100x ingest, 1x storage" model.

Many production systems use this: cheap to collect spans (5-10 GB/sec at scale isn't a lot of network), expensive to store them long-term, so the buffer + decide pattern wins.

Per-service sampling (the "bait and switch")
A service can sample at 1% locally but propagate the decision to children. If the parent decides "keep this trace", every downstream service keeps it too. Without this, you get partial traces (parent kept, child dropped) which are often worse than no trace.

OpenTelemetry's W3C trace context propagation handles this, but every service must respect the sampling bit.

Exemplars: the bridge between sampled trace and unsampled metric
Even with 1% trace sampling, every metric is computed from 100% of requests. The bridge: when you increment a metric, occasionally attach an exemplar (a sampled trace ID + the value).

When an engineer sees a latency spike on a metric, the exemplar lets them jump to a trace that contributed to the spike - even though they didn't predict which traces would matter.

This pattern (Prometheus exemplars + Tempo) has reshaped how teams debug latency regressions. It's the practical answer to "we sample 1% of traces but we want every alert to surface a relevant trace".

Cost tuning

• Default head-sample rate: 1-10% depending on service.
• Tail rules: keep all errors, all >p99 traces, all sampled-by-rule traces.
• Exemplar rate: 1% per metric bucket, capped at N exemplars per minute per metric.

Engineers should be able to escalate sampling for a specific service / endpoint during incident or performance work, then drop back. The platform makes this a self-serve knob.

The TSDB is the engine of the metrics signal. Choosing wrong locks the platform into the wrong cost curve.

Prometheus (single-node)
The reference implementation. PromQL is the de-facto query language. Single-binary, simple to operate, beloved by SREs.

Pros: best PromQL ergonomics. Strong community. Great pull-based scrape model.
Cons: single-node. Caps at ~10M active series. No long-term storage; no replication; no native HA.

Use when: <100 services, single team, <30-day retention. Outgrown by most companies past Series B.

M3 (Uber)
Sharded, replicated TSDB built on M3DB. Scales to billions of series. Used by Uber, Chick-fil-A, others.

Pros: proven at scale. Tiered storage. Multi-DC replication.
Cons: operationally heavy (Cassandra-class complexity). Smaller community than alternatives.

Use when: massive scale + dedicated platform team with distributed-systems chops.

VictoriaMetrics
Drop-in Prometheus replacement with much better resource usage. Both open source and commercial editions.

Pros: 5-10x lower memory than Prometheus at the same load. Single-node and clustered modes. Strong write throughput.
Cons: Smaller ecosystem than Prometheus + Thanos.

Use when: cost-driven, want simple ops, mid-scale (10s of millions of series).

Mimir / Cortex (Grafana ecosystem)
Horizontally scalable Prometheus-compatible TSDB. Mimir is the modern fork, Grafana-Labs maintained.

Pros: cloud-native architecture (object storage + ingester + query). Multi-tenant native. Active development.
Cons: many moving parts (ingester, distributor, store-gateway, compactor, querier). Object-storage-backed query latency.

Use when: large multi-tenant platform, heavy use of Grafana stack, comfortable with Kubernetes.

Thanos
Adds long-term storage + global query to vanilla Prometheus. Sidecar pattern.

Pros: incremental adoption from existing Prometheus.
Cons: query performance degrades at very high cardinality. Operational complexity grows.

Use when: existing Prometheus footprint, need to extend retention without re-platforming.

Cloud-native options

• AWS Managed Prometheus: hosted Cortex, pay-per-sample. Convenient, expensive at scale.
• Google Managed Service for Prometheus: similar. Tight integration with GKE.
• Datadog / New Relic: full SaaS, includes other signals. Easy to start, expensive to keep.

The decision framework

• Below 10M series: Prometheus + remote storage.
• 10M-100M series: VictoriaMetrics or Mimir.
• 100M+ series: M3, Mimir cluster, or managed offering.
• Cost-driven: VictoriaMetrics wins on $/series.
• Ops-burden-minimized: cloud-managed wins.

The migration cost
Switching TSDBs is months of work + dual-write transition + dashboard re-validation. Choose for 3-year scale, not 6-month scale.

Logs are the highest-volume signal. At 100 TB/day, the cost difference between architectures is millions of dollars per year.

The traditional model: Elasticsearch
Index every field at write time. Subsecond query on any field. The Splunk / ELK model.

Pros: fast ad-hoc query. Full-text search.
Cons: index size = 1-3x the raw data. Memory-hungry (heap + page cache). Operational complexity: shard rebalancing, hot/cold tiers, frequent OOMs.

Cost: ~$1-3 per GB ingested at scale. 100 TB/day = $100K-$300K/day = $36M-$110M/year. Untenable.

The new model: Grep-on-object-storage (Loki, VictoriaLogs)
Don't index. Store logs compressed in object storage. Query by filtering on metadata + brute-force grep on selected files.

Pros: ~10-100x cheaper storage (object vs SSD + index). Simpler ops.
Cons: queries can be slow (10s-minutes for broad searches). Requires good metadata to narrow the search space.

The trade: cheap to store, slower to search. Works well when most logs are written and never read; the rare query pays for the slowness.

The hybrid: Vector + S3 + Athena / Trino
Pipeline logs through a router (Vector, Fluent Bit) to S3 in Parquet format. Query with Athena or Trino.

Pros: cheap storage. SQL queries.
Cons: query latency 10s of seconds. Schema migrations are work.

Indexing selectively
Index high-value fields (request_id, user_id, error_code) for fast lookup. Leave the body unindexed. Query: "find request_id=ABC" is fast (uses index) → "show me the body" is fast (the index points to the location).

ClickHouse is increasingly popular for this: columnar storage with selective indexes, fast on both filter + aggregate.

Sampling logs
Often controversial. Two perspectives:

• "Logs are the source of truth; never sample." - misses cost reality.
• "Sample like you sample traces." - what most cost-disciplined teams do.

Pragmatic: sample debug-level by 90% (you rarely need it); keep 100% of warn/error. Keep all logs from canary deploys. Sample less aggressively for the most-recent N days; sample more aggressively for the rest.

Structured vs unstructured
JSON logs win. Unstructured logs require parsing at query time, lose information, and are slower to filter.

The hardest battle: convincing teams to change their logging libraries. Tools (logfmt, structlog, Zerolog, slog) make it easy enough that there's no excuse. Platform team enforces via CI lint.

Retention tiers

• Hot (7-30 days, indexed): SSD-backed. Fast query.
• Warm (30-90 days, object storage with selective indexes): seconds-to-minutes query.
• Cold (1+ year, Glacier-class): hours to retrieve. For compliance only.

Lifecycle policy moves data automatically. Engineers must understand which tier they're querying - the platform UI shows it explicitly.

OpenTelemetry (OTel) is the CNCF standard for instrumentation. Within five years, every observability platform will be OTel-native; resisting is a rear-guard action.

What OTel provides

• A vendor-neutral instrumentation API (metrics, logs, traces).
• A vendor-neutral wire protocol (OTLP).
• A vendor-neutral collector (the OTel Collector) that receives, transforms, and exports to any backend.

The "before" world
Each observability vendor had its own SDK. Switching from Datadog to New Relic meant re-instrumenting every service. Vendor lock-in was structural.

The "after" world
Instrument once with OTel. Send to whatever backend you want. Switch backends with a config change.

The collector pattern
The OTel Collector is a stateless agent (or gateway) that:

• Accepts OTLP from instrumented apps.
• Applies transformations: drop labels, sample, batch, enrich.
• Exports to any backend (Prometheus remote_write, Tempo, Loki, Datadog API, etc.).

This decouples the application from the backend. Migrate from one TSDB to another? Update the collector exporter; no app changes.

Auto-instrumentation
For Java, .NET, Python, Node.js, OTel offers auto-instrumentation - attach a Java agent / module loader, get HTTP / DB / messaging traces with zero code changes. Coverage is increasingly broad.

Migration strategy

1. Deploy OTel Collector alongside existing agents (Datadog agent, Prometheus scrape, etc.). Send same data to both.
2. Convert services to OTel SDK incrementally. Old services keep their existing instrumentation; new services start with OTel.
3. After a year of dual-write, retire the old agent.

The migration is ~12-24 months for a large org. Front-loading the cost gives back vendor flexibility for the next decade.

OTel for logs
Newest signal type in OTel. Spec is stable but tooling is less mature than metrics/traces. Most teams continue with their existing log shipper (Vector, Fluent Bit) and integrate via OTel later.

The collector as a control plane
A mature OTel Collector deployment becomes the platform's control plane:

• Cardinality limits enforced at the collector.
• Sampling rules configured centrally.
• Routing rules per signal type.
• PII redaction.

This consolidates policy enforcement that previously lived in dozens of agent configs.

Performance considerations
OTel SDKs add 1-5% CPU overhead at typical instrumentation volume. The collector adds 0.5-1% latency on the network path. Both are small relative to the operational benefit.

The interview signal
Candidates who default to "we'd use Datadog" without considering OTel signal lower than candidates who explain when each makes sense. OTel + self-hosted backend is the standard playbook for cost-driven orgs at scale; managed (Datadog / Honeycomb / Lightstep) is faster to ramp.

An alerting system that pages too often is ignored. An alerting system that misses incidents is useless. The math of alert design is harder than it looks.

The basic loop
Evaluate rules on a schedule (every 30s for critical, every 5m for non-critical). Each rule produces a series of "firing/not-firing" decisions. Transitions to firing trigger notifications.

Deduplication
A flapping condition (firing → resolved → firing → resolved) shouldn't page on every transition. Standard pattern:

• Sustain: the condition must hold for N intervals before alerting (e.g., 3 minutes).
• Hysteresis: the resolved threshold is lower than the firing threshold.
• Group by: similar alerts batched into one notification.

Alertmanager (Prometheus ecosystem) implements all three. Most engines copy the model.

Routing
Alerts route to teams based on labels (service, team, severity). Critical alerts page; warnings go to a Slack channel; info goes to a dashboard.

The routing tree must be code-reviewed. Teams should not be able to silently route their pages to /dev/null.

Silences
During a planned maintenance, alerts for the affected service get silenced. Silences must:

• Have a TTL (auto-expire).
• Require justification.
• Be auditable.

The "silenced for 30 days because it's noisy" pattern is the tell of a degrading alerting culture.

Alert types

• Threshold: simple "metric > X". Brittle - thresholds drift.
• Anomaly: ML-based or statistical (3 std-dev, EWMA). Smarter but harder to debug.
• Multi-condition: "p99 > 1s AND error rate > 0.1%". Reduces false positives.
• SLO burn rate: "we've burned 10% of our error budget in the last hour". Aligns with SRE practice; reduces false positives sharply.

SLO burn-rate alerting is the modern best practice. Ties alerts to user-perceived SLOs, not arbitrary thresholds.

Page volume targets

• Critical pages should resolve to <1 per on-call shift on average. >5/shift = burnout.
• Warning channel: should be skimmed, not acted on. <50/day per service.

If alerts exceed these, the alerting story is broken. Prune ruthlessly.

The alert lifecycle
Each alert should have:

• Runbook link (what to do when paged).
• Owner (who owns this alert).
• Last firing date (stale alerts that never fire are candidates for deletion).
• Last edit date (alerts that haven't been touched in years drift from current behavior).

Auditing alerts quarterly is platform team work. Automation helps; replacement humans are the bottleneck.

Self-monitoring of the alerting system itself
The alerting engine must have its own monitoring (separately deployed). When alerting goes silent for 5 minutes, that's the worst incident - not just service down, but no one knows.