We use cookies for site analytics. Accept to help us understand how the site is used. See our Privacy Policy for details.
The front door to a microservice fleet. One layer that every external request crosses - so it owns auth, rate limiting, routing, and resilience, and absolutely cannot become the bottleneck.
Design an API gateway: the single entry point that sits in front of a fleet of backend microservices and handles the cross-cutting concerns every service would otherwise reimplement - TLS termination, authentication, authorization, rate limiting, routing, request/response transformation, and observability.
This is a Hard system design topic because the gateway is on the critical path of 100% of external traffic. Its design is a study in keeping a shared, stateful-ish layer fast, horizontally scalable, and resilient - and in cleanly separating the high-throughput data plane from the control plane that configures it.
Asking these before diving into a solution is the difference between a "hire" and a "no signal" rating. Pick the questions whose answers would change your design.
Throughput and fleet size
CPU is the bottleneck, not bandwidth
Rate-limit state
Config size
The defining split is data plane vs control plane. The data plane is a fleet of stateless proxy nodes that process requests at line rate through a pipeline of filters (TLS → auth → rate limit → route → transform → forward). The control plane is a separate, low-throughput service where operators define routes/policies/keys; it validates them and pushes versioned snapshots to every data-plane node. Requests never touch the control plane, so config operations can never slow down or take down request serving.
Spreads incoming connections across the stateless gateway fleet and across AZs. The gateway itself is L7; this layer just distributes connections and removes dead nodes.
Run the per-request filter chain: terminate TLS, authenticate, authorize, rate-limit, match route, optionally transform, forward to the backend, and emit telemetry. Hold config in memory; scale horizontally.
Validates API keys and JWTs. JWT signatures are checked locally against a cached JWKS (public keys), so the common path needs no network call. Opaque tokens/introspection are cached aggressively with short TTLs.
Local token buckets per node for the hot path, periodically reconciled against a shared store (Redis/DynamoDB) for a globally coordinated approximate limit. Fail-open for protective limits, fail-closed for billing-critical ones.
Resolves the matched route to a healthy backend instance set via a service registry (Consul/etcd/EDS) or DNS, with active health checking and outlier ejection.
Operator-facing API to manage routes, policies, consumers, and keys. Validates changes, versions them, and streams updates to all data-plane nodes (xDS-style push). Never on the request path.
Per-request structured logs, RED metrics (rate/errors/duration), and distributed traces shipped async so telemetry never blocks request serving.
The subsystems where the interview is actually decided. Skim if you're running short; own these if you want a strong signal.
Model the gateway as an ordered pipeline of filters; each can short-circuit. Ordering is a correctness decision, not a style choice.
A typical chain:
Why order matters: authenticate before rate-limiting so limits are per-consumer, not per-IP. Match the route before authenticating so unmatched garbage is cheap to reject. Cheap rejections go early; expensive work goes late.
Interview signal: candidates who describe the gateway as a configurable, ordered filter chain (rather than a monolith of if-statements) demonstrate they've thought about extensibility and the data/control-plane split.
Auth on the hot path must not become a synchronous call to an auth service per request - that adds latency and couples the gateway's availability to the auth service's.
JWT (stateless tokens): the gateway validates the signature locally using the identity provider's public keys (JWKS), which it caches and refreshes periodically. Zero network calls on the common path. The token carries claims (subject, scopes, expiry); the gateway checks expiry and required scopes. Trade-off: revocation is hard - a stolen token is valid until it expires, so keep token lifetimes short and pair with a denylist for emergencies.
API keys / opaque tokens: must be looked up to resolve identity and plan. Cache the key→consumer mapping aggressively (short TTL) so the common path is a local hit; only cache misses touch the store.
mTLS: for service-to-service or high-trust callers, the client certificate is the identity - validated during the TLS handshake, no token needed.
Authorization is a separate step: authentication says who you are, authorization says what you may do. Keep policy declarative (route → required scopes/roles) and evaluated locally from config, not fetched per request.
The key principle: push verification to local computation against cached material. Network calls in the auth path are an availability and latency tax you pay on 100% of traffic.
Rate limiting is the gateway's signature feature and its trickiest piece of shared state, because the fleet is many nodes but limits are global ("1000 req/min for this consumer across all of them").
Naive central counter: every node does a Redis INCR per request. Correct, but adds a round trip (1-2ms) to every request and makes Redis a throughput ceiling and a SPOF.
Local token buckets + reconciliation (the standard answer): each node enforces a local bucket sized to its share of the global limit, and nodes periodically sync usage to a central aggregator that rebalances budgets. Hot path is a local, lock-free decision; the central store sees only periodic batched updates. Cost: small inaccuracy at window boundaries - acceptable for protective limits, tightened for billing.
Layered limits: a request can match several rules at once - global, per-consumer, per-route, per-IP. Evaluate all applicable rules; the strictest denial wins. Return standard headers (X-RateLimit-Remaining, Retry-After) so well-behaved clients back off.
Fail-open vs fail-closed: if the rate-limit store is unreachable, protective limits should fail open (don't turn a Redis blip into a full outage), but billing-critical limits fail closed. Choose per rule, not globally.
This connects directly to the standalone Rate Limiter design - the gateway is its most common deployment home.
The gateway concentrates risk: it's on the path of every request, so its failure modes are everyone's failure modes. The design must actively prevent it from amplifying backend problems.
Stateless, redundant data plane: nodes hold no per-request state, so any node can serve any request and you run N+ behind an L4 LB across AZs. A dead node is just removed from rotation.
Protect the backends from the gateway:
Protect the gateway from clients: rate limiting and request-size limits at the door; shed load gracefully (503 + Retry-After) under overload rather than collapsing.
Zero-downtime config reloads: apply new route/policy versions atomically and drain old connections, so a config push never drops in-flight requests.
Interview signal: the strongest answers treat the gateway as a resilience layer that isolates failures, not just a router - circuit breaking, retry budgets, and bulkheads show you understand it can make an outage worse if designed naively.
Centralized cross-cutting concerns vs per-service duplication
A gateway means auth, rate limiting, and observability are implemented once instead of in every service - huge consistency win. The cost is a shared layer on the critical path that every team depends on and that can become an organizational bottleneck if one team owns all config.
Data plane vs control plane separation
Splitting them keeps request serving fast and isolated from config churn, and lets each scale independently. The cost is added architectural complexity and a config-propagation pipeline you must build and operate.
Local-cache auth/limits vs strong correctness
Validating tokens locally and using local rate-limit buckets keeps the hot path fast and decoupled, at the cost of weaker revocation and small limit inaccuracy. For 100% of traffic, that trade almost always favors local computation.
Thin gateway vs fat gateway
A thin gateway (route + auth + limit) stays fast and simple; a fat one (heavy transformation, aggregation, business logic) becomes a distributed monolith that's hard to evolve and a deployment chokepoint. Keep business logic in services; keep the gateway about cross-cutting concerns.
Single gateway vs BFF-per-client
One general gateway is simplest; a Backend-for-Frontend per client type (web/mobile/partner) tailors payloads and limits but multiplies the layers to operate. Start with one; split to BFFs when client needs genuinely diverge.
Be ready for at least three of these. The first one is almost always asked.
Five algorithms, three sharding strategies, one fail-open vs fail-closed decision. The bounded design that surfaces in every backend interview loop.
L4 vs L7, consistent hashing, health checks, connection draining, and the difference between a fleet that survives partial failures and one that cascades into outage.
The internet's phone book. A globally distributed, read-dominated, cache-everywhere hierarchy that has to answer in single-digit milliseconds and never go down.
Reading is the floor. The interview signal is in walking through this live with someone probing follow-ups. Use the AI mock interview to practice talking through requirements, architecture, and trade-offs out loud.
Start an AI mock interview →