What's the traffic profile - QPS, request size, sync REST vs streaming/WebSocket vs gRPC?

100K-1M+ requests/sec across a horizontally scaled gateway fleet. Stateless data-plane nodes behind an L4 load balancer / anycast, scaling out linearly with traffic.

What are the latency budget and availability targets the gateway must not violate?

The gateway adds overhead to every request, so its own processing budget is tight: p99 added latency < 5-10ms excluding the backend call. Auth checks and rate-limit lookups must be near-instant (cached/local), never a synchronous DB round-trip on the hot path.

Networking

Design an API Gateway

Hard Premium

The front door to a microservice fleet. One layer that every external request crosses - so it owns auth, rate limiting, routing, and resilience, and absolutely cannot become the bottleneck.

The problem

Design an API gateway: the single entry point that sits in front of a fleet of backend microservices and handles the cross-cutting concerns every service would otherwise reimplement - TLS termination, authentication, authorization, rate limiting, routing, request/response transformation, and observability.

This is a Hard system design topic because the gateway is on the critical path of 100% of external traffic. Its design is a study in keeping a shared, stateful-ish layer fast, horizontally scalable, and resilient - and in cleanly separating the high-throughput data plane from the control plane that configures it.

Clarifying questions

Asking these before diving into a solution is the difference between a "hire" and a "no signal" rating. Pick the questions whose answers would change your design.

→What's the traffic profile - QPS, request size, sync REST vs streaming/WebSocket vs gRPC?
→Which cross-cutting concerns are in scope: auth, rate limiting, routing, transformation, caching, retries?
→What auth schemes must we support - API keys, JWT/OAuth bearer tokens, mTLS?
→How are backend services discovered - static config, DNS, or a service registry?
→Do we need per-route, per-consumer, and global rate limits simultaneously?
→Is this north-south only (external → internal), or also east-west (service-to-service)?
→What are the latency budget and availability targets the gateway must not violate?
→Who configures routes/policies and how often do they change - do we need zero-downtime config reloads?

Requirements

Functional requirements

·Terminate TLS and accept external requests on a stable set of endpoints
·Authenticate the caller (API key / JWT / mTLS) and authorize the requested route
·Route the request to the correct backend service via path/host/header matching
·Enforce rate limits and quotas (global, per-consumer, per-route)
·Optional: request/response transformation, response caching, retries with backoff, circuit breaking
·Emit structured logs, metrics, and traces for every request
·Hot-reload routes and policies with no dropped connections

Non-functional requirements

Scale: 100K-1M+ requests/sec across a horizontally scaled gateway fleet. Stateless data-plane nodes behind an L4 load balancer / anycast, scaling out linearly with traffic.
Latency: The gateway adds overhead to every request, so its own processing budget is tight: p99 added latency < 5-10ms excluding the backend call. Auth checks and rate-limit lookups must be near-instant (cached/local), never a synchronous DB round-trip on the hot path.
Availability: 99.99%+. The gateway is a hard dependency for every API consumer; if it's down, the whole API surface is down. No single point of failure - N+ redundant stateless nodes, multi-AZ.
Consistency: Config (routes, policies, keys) is eventually consistent - a route change propagates to all nodes within seconds. Rate-limit counters are approximately consistent (small over/under-count under partition is acceptable; exact billing limits get stronger handling).

The full walkthrough covers

Architecture components

·L4 load balancer / anycast
·Gateway data-plane nodes (stateless proxies)
·Auth/identity integration
·Rate-limit subsystem
·Service discovery / routing table
·Control plane + config store
·Observability pipeline

Deep dives

The request lifecycle as a filter chain
Authentication and authorization without a round trip
Distributed rate limiting at the gateway
Resilience: not becoming a single point of failure

What's in the full walkthrough

Capacity estimation · architecture with all 7 components explained · 4 deep dives · trade-off analysis · 8 common follow-up questions

This system design walkthrough is part of Premium

Unlock it with a subscription.

Start Your 7-Day Trial

Try everything free for 7 days. Cancel anytime.

Every system design walkthrough - capacity estimates, deep dives and trade-offs
Full-length AI mock interviews - 3 to 10 questions with follow-ups, up to 3 a day

View Plans

7-day trial, then $8/mo - or $48/yr ($4/mo, save 50%). Cancel anytime.

Companies that test this topic

Practice in interview format

Reading is the floor. The interview signal is in walking through this live with someone probing follow-ups. Use the AI mock interview to practice talking through requirements, architecture, and trade-offs out loud.

Mock interview: design API Gateway →