Two years ago, "design an ML system" meant feature stores and training pipelines. In 2026, the AI system design round has converged on something much more specific: can you design the serving and retrieval infrastructure behind an LLM product? Inference platforms, RAG pipelines, semantic search, copilots - systems where the expensive, failure-prone component is a model, and the trade-offs are measured in tokens, GPU-hours, and hallucination rates.
If you're interviewing for AI engineer, ML engineer, or platform roles at any company with an LLM feature (which is most of them), expect at least one round like this. (Not sure which of those roles you're targeting? Our AI engineer vs. ML engineer vs. MLOps breakdown maps who owns what.)
This guide covers the questions that actually come up, organized the way the rounds run: the big design prompts first, then the cost/latency follow-ups interviewers use to probe depth, then evaluation. If you want the classic fundamentals first, start with our top 30 system design questions - everything there still applies here.
How AI system design rounds differ from classic ones
Worth internalizing before the questions, because interviewers grade against these differences:
-
Cost is a first-class requirement, not a footnote. In a classic round, you can hand-wave "we'll add servers." In an AI round, tokens and GPU-hours are the dominant line item, and interviewers expect cost reasoning threaded through every decision - model choice, caching, routing, context size. "How much does this cost per request?" is a standard follow-up.
-
The bottleneck is inference, not the database. Classic rounds are won with caching, sharding, and queues. Here, model latency dwarfs everything else in the request path, so the levers change: KV cache management, batching strategy, streaming, prompt caching, model routing. Your classic toolkit still matters for everything around the model - it's just no longer where the round is decided.
-
Output quality is probabilistic, so evaluation is architecture. A classic system either returns the right rows or it doesn't. An LLM system returns something every time, and whether it's good is a measurement problem. Strong candidates design the eval and monitoring loop as part of the system, unprompted.
-
Requirements clarification has new axes. Beyond QPS and latency: What's the token budget per request? Is quality or cost the priority? Can we use a hosted API, or is this self-hosted for privacy? Is stale data acceptable, and for how long? Asking these early signals you've built these systems.
The round's shape is unchanged - requirements, estimation, high-level design, deep-dives, trade-offs. Our system design walkthroughs drill that structure topic by topic, including the LLM inference and RAG topics that map directly onto this post.
Section 1: Design an LLM inference platform
The flagship prompt, in several disguises: "Design ChatGPT," "Design the serving layer for our AI assistant," "Design a multi-tenant LLM API."
Q1: Walk me through the high-level architecture.
The skeleton interviewers expect:
- API gateway - auth, per-tenant rate limiting, request validation
- Router - picks a model (and hardware pool) per request: small/cheap model for simple requests, large model for complex ones
- Request queue + scheduler - admission control and batching; this is where fairness and priority live
- Inference workers - GPU nodes running a serving engine (vLLM-style continuous batching is the reference answer in 2026)
- Streaming layer - tokens stream back over SSE/WebSockets as they're generated
- Caching tiers - prompt/prefix cache at the serving engine, optional semantic response cache in front
- Observability + billing - per-request token metering, latency percentiles split by phase, quality sampling
The differentiator isn't naming the boxes - it's knowing which two metrics rule the design: time-to-first-token (TTFT), which is the user's perceived responsiveness, and inter-token latency / total throughput, which is your GPU bill. Almost every design decision trades one against the other.
Q2: Why is LLM inference hard to serve efficiently?
The answer interviewers want is the two-phase structure:
- Prefill - processing the input prompt. Compute-bound, parallel across the whole prompt, determines TTFT.
- Decode - generating tokens one at a time. Each step depends on the last, is memory-bandwidth-bound, and leaves GPU compute badly underutilized if you serve one request at a time.
Add the killer constraint: requests are wildly heterogeneous. A 50-token question and a 100k-token document summary hit the same fleet. Naive request-level batching means short requests wait for long ones; no batching means single-digit GPU utilization. That tension is what the whole serving stack exists to resolve.
Q3: How do you scale it and handle multi-tenancy?
Talking points that land:
- Separate prefill-heavy and decode-heavy pools (or use engines with chunked prefill) so long prompts don't starve interactive chats
- Autoscale on queue depth and KV-cache memory pressure, not CPU - GPU fleets have slow cold starts, so keep a warm pool and scale predictively on traffic patterns
- Per-tenant token budgets and priority tiers enforced at the scheduler, so one tenant's batch job can't torch everyone's TTFT
- Graceful degradation - under load, route to smaller models, cap max output tokens, and shed lowest-priority traffic first, rather than queueing into timeout
Section 2: Design a RAG pipeline
The second-most-common prompt: "Design a system that answers questions over our internal documents."
Q4: What are the components, end to end?
Two planes:
- Indexing (offline): ingest -> chunk -> embed -> store in a vector index with metadata. Design it as a pipeline with change-data-capture from source systems, so updated documents re-index incrementally instead of via full rebuilds.
- Query (online): embed query -> retrieve candidates (hybrid: dense + keyword, merged) -> rerank to top-k -> assemble prompt -> generate -> cite sources.
In a system design round (versus an ML round), the scoring weight shifts to the systems concerns: index freshness SLAs, permission filtering (a user must never retrieve documents they can't read - enforce ACLs in the retrieval query, not post-hoc), multi-tenancy isolation, and what happens when retrieval returns nothing good.
We keep a separate deep-dive on the ML side of this - chunking, embeddings, rerankers, agentic RAG - in our RAG interview questions guide; this round cares that you can wrap those choices in production infrastructure.
Q5: How do you keep the index fresh, and what does staleness cost?
Strong answers name a freshness SLA ("edits visible in search within 15 minutes") and derive the architecture from it: CDC or event streams from source systems, an incremental indexing queue, tombstoning deleted docs immediately (stale presence of a deleted doc is usually a worse failure than a missing update), and a periodic full re-index path for when the embedding model changes - noting that swapping embedding models invalidates the entire index, so you shadow-build the new one and cut over.
Q6: The retriever returns garbage for some queries. Where do you look?
A debugging question disguised as design. The systematic answer: instrument the pipeline so you can tell retrieval failure (right doc never in the candidate set - a chunking/embedding/hybrid-weighting problem) from ranking failure (right doc retrieved but reranked out of the top-k) from generation failure (right context in the prompt, wrong answer out). Each has different fixes, and saying "I'd log retrieved chunks with scores per query so I can replay failures" is worth more than naming any specific fix.
Section 3: Design semantic search
Sometimes standalone ("design search for a product catalog / help center"), sometimes a RAG sub-round.
Q7: Dense, sparse, or hybrid - and how do you serve it at scale?
The converged 2026 answer is hybrid: dense embeddings for meaning, BM25-style keyword for exact terms (SKUs, error codes, names), merged with reciprocal rank fusion, then a cross-encoder reranker over the top 20-50 candidates. The system design layer on top:
- ANN index choice - HNSW for recall/latency at the cost of memory; IVF/quantization when the corpus is huge and memory-bound. Know that this is a recall-vs-memory-vs-latency triangle
- Sharding the index by tenant or document space, with metadata filtering pushed down into the index rather than post-filtering
- The reranker is the latency hot spot - it's a model call per candidate set, so it gets its own pool, batching, and a latency budget; under pressure you cut candidate count before you cut the reranker entirely
Q8: How is this different from designing classic keyword search?
A compare-and-contrast question testing whether you know what stayed the same. Same: inverted-index-era concerns like sharding, replication, freshness, filters. New: an embedding service in both the write and read paths (a new dependency that can drift between versions), ANN instead of exact lookup (recall becomes a tunable, which classic search never had), and evaluation - you now need labeled relevance data to know whether search got better, because there's no exact-match ground truth.
Section 4: The cost/latency follow-ups
These are the probes interviewers drop into any of the designs above to separate candidates who've operated these systems from candidates who've read about them.
Q9: What is the KV cache and why does it dominate GPU memory?
During decode, the model reuses attention keys/values for every token already processed; caching them avoids recomputing the whole prefix per generated token. The cost: the cache grows linearly with context length and batch size, and at long contexts it - not the model weights - is what limits how many requests fit on a GPU. That makes KV-cache memory the real currency of an inference fleet. Follow-ups to be ready for: paged KV-cache allocation (vLLM's core idea - fixed-size blocks instead of contiguous reservations, cutting fragmentation), and why "just raise max context length" is a capacity decision, not a config flag.
Q10: Explain batching strategies and their trade-offs.
- No batching - best per-request latency, terrible GPU utilization; only defensible for a latency-critical single-tenant path
- Static batching - wait to fill a batch, run it to completion. Throughput up, but short requests get held hostage by the longest one, and utilization craters as sequences finish at different times
- Continuous (in-flight) batching - the 2026 default: new requests join the running batch at token-step granularity, finished ones exit immediately. Near-static-batching throughput with far better tail latency
The judgment layer: batching is a throughput-vs-TTFT dial. Interactive chat wants small effective batches and streaming; offline summarization wants the batch as fat as memory allows. Strong answers propose separate queues/pools per workload class rather than one compromise setting.
Q11: Where does caching help, and what are the layers?
Three distinct layers - naming all three cleanly is a strong signal:
- Prompt/prefix caching - reuse prefill computation for shared prompt prefixes (system prompts, few-shot examples, RAG boilerplate). Hosted APIs price cached input tokens at a steep discount, so structuring prompts so the stable part comes first is a real architectural decision, not a nicety
- Semantic response caching - serve a previous answer for a semantically-equivalent query. Big win on skewed traffic; risky where answers are personalized or freshness-sensitive, so scope it and TTL it
- Classic caching - embeddings for repeated queries, retrieval results, rendered context blocks - the same first-lever thinking as any design round
Q12: Your inference bill doubled. Walk me through cutting it without wrecking quality.
An operations question in design clothing. A credible sequence: measure first (cost per request by feature and by tenant - you can't cut what you can't attribute), then in rough order of pain-to-payoff: prompt caching and prompt slimming (input tokens usually dwarf output), model routing (a classifier sends the easy majority of traffic to a small model, hard cases to the big one), output caps and semantic caching, batching/utilization tuning, and only then quantization or model swaps - which change output behavior and therefore require the eval suite from Section 5 before rollout. Bonus points for naming the guardrail: per-feature token budgets with alerts, so the next doubling pages someone before the invoice does.
Section 5: Evaluation and hallucination questions
Q13: How do you evaluate an LLM system before and after launch?
The framework that lands: offline evals (a golden set of prompts with graded expected behavior, run on every prompt/model change, gating deploys the way tests gate merges), online sampled evals (an LLM-judge scoring a slice of production traffic for faithfulness/relevance, calibrated against a human-labeled subset - and pin the judge model's version, or your dashboards drift when it changes), and product signals (thumbs, regenerations, escalations to a human). The architectural point: this is a pipeline you design and operate - eval sets are versioned artifacts, and every prompt change gets an eval run attached.
Q14: How do you reduce hallucination architecturally?
Move it from "model problem" to "system problem": ground answers via retrieval and require citations; constrain output with schemas/structured outputs where the product allows; give the system an explicit "I don't know" path when retrieval confidence is low (returning nothing is a feature, not a failure); add a verification pass for high-stakes outputs (a second model checks claims against the retrieved context); and monitor faithfulness continuously via the sampled evals above. Interviewers reward candidates who say hallucination rate is an SLO you measure and budget against, not a bug you fix once.
Q15: A prompt change improved your golden set but users are complaining. What happened?
Eval-set overfitting or coverage gaps - the golden set no longer represents production traffic. The mature answer covers: continuously refresh the eval set from sampled (and anonymized) production queries, keep a held-out set that prompt authors never see, segment complaints to find the failing slice, and treat "eval says fine, users say broken" as a monitoring gap to close, not a user-error to dismiss.
How to prep for this round
The failure mode to avoid is memorizing this post the way dump-users memorize exams. These rounds reward the same muscle as classic design - reasoning out loud from requirements to trade-offs - with a new vocabulary of costs. To build it:
- Run the full loop on each prompt above, timed, out loud: requirements, estimation (tokens per request, requests per second, GPUs needed - rough numbers beat no numbers), architecture, two deep-dives, trade-offs.
- Study the walkthroughs. Our system design walkthroughs - including the LLM inference and RAG topics - model the structure interviewers grade against, with the capacity-estimation habit built in.
- Follow a path if you're ramping. The learning paths sequence system design alongside the ML fundamentals these rounds assume - the ML engineer and cloud architect paths are the closest fits.
- Pressure-test in a mock. A timed AI mock interview in system design mode will probe your KV-cache answer with follow-ups the way a real interviewer will, which is where reading stops helping and practice starts.
The candidates who pass these rounds in 2026 aren't the ones who know the most acronyms. They're the ones who treat an LLM like what it is in a system diagram: an expensive, slow, occasionally-wrong dependency - and design around it accordingly.
Good luck. Go design something.