Latency target - interactive streaming (time-to-first-token matters) or async/batch (throughput matters)?

Time-to-first-token (TTFT) p50 < 800ms, p99 < 2.5s including retrieval. Inter-token latency p50 < 40ms so text reads as it streams. Retrieval budget: < 150ms for search + rerank.

gitGood.dev

Distributed Systems

Design an LLM Inference / RAG Platform

Hard Premium

Prefill vs decode, paged KV cache, prompt caching, vector search + reranking, groundedness evals, and why you autoscale on queue depth measured in tokens - not requests.

The problem

Design the serving platform behind an AI assistant product: users ask questions, the system retrieves relevant documents from a private corpus (RAG), assembles a prompt, runs it through a large language model on a GPU fleet, and streams the answer back token by token with citations.

This is the AI-era version of "design a web service" and it's rapidly becoming a standard question for AI-infra and senior backend loops. It layers two hard problems: an inference serving layer where the unit of work is a token (not a request) and the bottleneck is GPU memory bandwidth, and a retrieval pipeline whose quality silently determines whether the model answers or hallucinates. Strong candidates separate prefill from decode, own the cost-per-token math, and treat retrieval quality as a measurable, eval-gated system - not a vibes-based prompt tweak.

Clarifying questions

Asking these before diving into a solution is the difference between a "hire" and a "no signal" rating. Pick the questions whose answers would change your design.

→Is this a chat product (multi-turn, stateful KV cache) or single-shot Q&A (stateless)?
→Self-hosted open-weights models, a managed API (OpenAI / Anthropic / Bedrock), or a hybrid?
→What's the corpus - size, update rate, access control? Public docs or per-tenant private data?
→Latency target - interactive streaming (time-to-first-token matters) or async/batch (throughput matters)?
→What model sizes - one flagship 70B-class model, or a router across small and large models?
→How accurate must answers be - is a wrong-but-fluent answer a UX annoyance or a compliance incident?
→Multi-tenancy - can tenants share GPU capacity, or do some require isolation?
→What's the cost envelope - is GPU spend the dominant line item we're optimizing?

Requirements

Functional requirements

·POST /chat: accept a conversation + question, stream the answer token-by-token (SSE)
·Retrieve relevant chunks from the corpus, cite sources in the answer
·Ingest documents continuously: parse, chunk, embed, index; updates visible within minutes
·Multi-turn conversations with context carried across turns
·Per-tenant document isolation - retrieval never crosses tenant boundaries
·Model/prompt versioning with eval-gated rollout and instant rollback
·Per-tenant rate limits and token budgets (cost containment)

Non-functional requirements

Scale: 50K DAU, 2K concurrent sessions at peak, ~70 requests/sec peak. Corpus: 10M document chunks, 50K new/changed documents per day. Average request: ~2K prompt tokens (question + history + retrieved context), ~300 output tokens.
Latency: Time-to-first-token (TTFT) p50 < 800ms, p99 < 2.5s including retrieval. Inter-token latency p50 < 40ms so text reads as it streams. Retrieval budget: < 150ms for search + rerank.
Availability: 99.9% for the serving path. Degrade gracefully: if retrieval is down, answer from model knowledge with a banner; if the primary model pool is saturated, shed to a smaller model or queue with a visible position.
Consistency: Corpus freshness is an SLA (minutes from ingest to retrievable), not strict consistency. Conversation state is per-session and sticky-routed. Billing counters must be accurate; everything else is best-effort.

The full walkthrough covers

Architecture components

·API gateway
·RAG orchestrator
·Inference scheduler / router
·GPU inference pool (vLLM / TensorRT-LLM / SGLang)
·Ingestion pipeline
·Vector + keyword index
·Reranker
·Eval + observability stack

Deep dives

Prefill vs decode: two workloads wearing one trench coat
KV cache and prompt caching: memory is the real GPU bottleneck
The RAG ingestion pipeline: chunking, embeddings, and indexes that don't rot
Vector search and reranking: recall is cheap, precision is expensive
Groundedness and evals: shipping prompt changes without flying blind
Streaming, SLOs, and autoscaling on queue depth (measured in tokens)

What's in the full walkthrough

Capacity estimation · architecture with all 8 components explained · 6 deep dives · trade-off analysis · 8 common follow-up questions

Subscribe to Continue

Get full access to gitGood.dev with a subscription.

Start Your 10-Day Trial

Try everything free for 10 days. Cancel anytime.

1,950+ practice questions
AI mock interviews (3/day)
Coding challenges with real execution
Verified skill badges
146 certification guides, 26 with practice exams

View Plans

10-day trial, then $5/mo - or $40/yr ($3.33/mo, save 33%). Cancel anytime.

Companies that test this topic

Practice in interview format

Reading is the floor. The interview signal is in walking through this live with someone probing follow-ups. Use the AI mock interview to practice talking through requirements, architecture, and trade-offs out loud.

Start an AI mock interview →

Distributed Systems

Design an LLM Inference / RAG Platform

Hard Premium

Prefill vs decode, paged KV cache, prompt caching, vector search + reranking, groundedness evals, and why you autoscale on queue depth measured in tokens - not requests.

The problem

Clarifying questions

Asking these before diving into a solution is the difference between a "hire" and a "no signal" rating. Pick the questions whose answers would change your design.

→Is this a chat product (multi-turn, stateful KV cache) or single-shot Q&A (stateless)?
→Self-hosted open-weights models, a managed API (OpenAI / Anthropic / Bedrock), or a hybrid?
→What's the corpus - size, update rate, access control? Public docs or per-tenant private data?
→Latency target - interactive streaming (time-to-first-token matters) or async/batch (throughput matters)?
→What model sizes - one flagship 70B-class model, or a router across small and large models?
→How accurate must answers be - is a wrong-but-fluent answer a UX annoyance or a compliance incident?
→Multi-tenancy - can tenants share GPU capacity, or do some require isolation?
→What's the cost envelope - is GPU spend the dominant line item we're optimizing?

Requirements

Functional requirements

·POST /chat: accept a conversation + question, stream the answer token-by-token (SSE)
·Retrieve relevant chunks from the corpus, cite sources in the answer
·Ingest documents continuously: parse, chunk, embed, index; updates visible within minutes
·Multi-turn conversations with context carried across turns
·Per-tenant document isolation - retrieval never crosses tenant boundaries
·Model/prompt versioning with eval-gated rollout and instant rollback
·Per-tenant rate limits and token budgets (cost containment)

Non-functional requirements

Scale: 50K DAU, 2K concurrent sessions at peak, ~70 requests/sec peak. Corpus: 10M document chunks, 50K new/changed documents per day. Average request: ~2K prompt tokens (question + history + retrieved context), ~300 output tokens.
Latency: Time-to-first-token (TTFT) p50 < 800ms, p99 < 2.5s including retrieval. Inter-token latency p50 < 40ms so text reads as it streams. Retrieval budget: < 150ms for search + rerank.
Availability: 99.9% for the serving path. Degrade gracefully: if retrieval is down, answer from model knowledge with a banner; if the primary model pool is saturated, shed to a smaller model or queue with a visible position.
Consistency: Corpus freshness is an SLA (minutes from ingest to retrievable), not strict consistency. Conversation state is per-session and sticky-routed. Billing counters must be accurate; everything else is best-effort.