Systems

LLMs & Prompt Engineering Cheat Sheet

SystemsFREELast updated: July 2026 · By gitGood Editorial

How LLMs work in one paragraph, the knobs (context window, temperature, top-p), system vs user prompts, few-shot and chain-of-thought, RAG and embeddings, the fine-tune-vs-prompt decision, tool calling, eval basics, and the interview questions teams actually ask now.

How LLMs work, in one paragraph

A large language model is a transformer neural network trained on enormous text corpora to do one thing: given a sequence of tokens (word fragments - roughly 3/4 of an English word each), predict a probability distribution over the next token. Generation is that step in a loop - sample a token, append it, predict again - which is why output streams word-by-word and why cost scales with tokens. Pre-training on next-token prediction produces a raw model that merely continues text; post-training (instruction tuning plus preference optimization like RLHF) shapes it into an assistant that follows instructions and declines bad requests. Two consequences drive everything else on this sheet: the model has no live knowledge beyond its training cutoff (hence RAG and tools), and it optimizes plausibility, not truth (hence hallucination and evals). It is stateless - each request stands alone, and 'memory' is just prior messages being resent in the context window.

Core knobs and terms

Token: The unit of everything: ~4 characters / ~0.75 English words. Pricing, context limits, and latency are all denominated in tokens (input and output priced separately, output usually costing more).
Context window: The maximum tokens the model can attend to per request - prompt + conversation history + retrieved documents + its own output, all sharing one budget. Long contexts cost linearly and degrade retrieval quality toward the middle ('lost in the middle') - stuffing the window is not a strategy; put critical instructions at the start and the freshest content near the end.
Temperature: Rescales the next-token distribution before sampling. Low (~0) -> near-greedy, consistent, best for extraction/classification/code; higher (~0.7-1.0) -> more varied, best for brainstorming and prose. Even temperature 0 is not perfectly deterministic in practice (batching/floating-point nondeterminism on served models).
Top-p (nucleus sampling): Sample only from the smallest set of tokens whose probabilities sum to p (e.g. 0.9), truncating the long unlikely tail. An alternative dial to temperature - adjust one, not both.
System vs user prompts: The system prompt is the developer's channel: role, rules, tone, output format, tool definitions - set once, models are trained to weight it above conversation. User messages carry the per-turn task. Keep the two separate and never paste untrusted input into the system prompt: that separation is your main lever against prompt injection.
Max output tokens / stop sequences: Hard caps on generation length, and strings that end generation early. A response cut off mid-sentence usually means the output cap hit - check finish_reason before blaming the model.

Prompting techniques that actually move quality

Be specific + structure the prompt: State the task, audience, constraints, and output format explicitly; delimit inputs with fences or XML-style tags so instructions and data cannot blur. Most 'the model is dumb' complaints are underspecified prompts.
Few-shot examples: Show 2-5 input->output pairs before the real input. The single most reliable technique for format fidelity and edge-case handling - examples beat descriptions. Keep them diverse and mind the token cost; the model will copy the examples' style, including their mistakes.
Chain-of-thought (CoT): Ask the model to reason step-by-step before answering ('think through this first, then give the final answer'). Big gains on math, logic, and multi-constraint tasks; the mechanism is that intermediate tokens give the computation somewhere to happen. Reasoning-tuned models do this internally - prompting an explicit scratchpad matters most on standard models.
Role / persona: 'You are a senior SRE reviewing a runbook' focuses vocabulary, priorities, and depth. Cheap and effective framing, not magic.
Structured output: For machine-consumed output, demand JSON matching a schema (or use the API's structured-output / JSON mode), give a field-by-field example, and validate + retry-on-parse-failure in code. Never regex prose and hope.
Decompose the task: One prompt doing extraction + judgment + formatting does each worse than three chained prompts doing one apiece. Pipelines of small prompts are easier to eval and debug than one mega-prompt.

RAG in a nutshell

Retrieval-Augmented Generation grounds the model in knowledge it was not trained on: at query time, retrieve the most relevant chunks from YOUR corpus and paste them into the prompt with instructions to answer from them (and to say so when they do not contain the answer). The offline half: split documents into chunks (typically a few hundred tokens, some overlap), embed each chunk into a vector, store vectors in an index. The online half: embed the user's question, nearest-neighbor search for the top-k chunks, optionally rerank them with a stronger model, build the prompt, generate - ideally with citations back to the chunks so answers are checkable. Why RAG instead of fine-tuning for knowledge: the corpus can change hourly without retraining, answers cite sources, and access control stays enforceable at the retrieval layer (the model only sees what this user may see). Where RAG fails in practice - and what interviewers ask about - is almost always retrieval, not generation: bad chunking that splits answers across chunk boundaries, pure semantic search missing exact identifiers (fix: hybrid dense + keyword search), or top-k returning near-duplicates. Evaluate retrieval (did the right chunk surface?) separately from generation (was the answer faithful to it?).

Embeddings

What they are: A model that maps text to a fixed-length vector (hundreds to thousands of dimensions) where semantic similarity becomes geometric closeness - 'refund policy' lands near 'money-back guarantee' despite zero shared words. Compared with cosine similarity, searched with a vector index.
What they power: Semantic search and RAG retrieval, clustering/dedup, recommendations, and classification-by-nearest-neighbor. Embedding models are separate from and far cheaper than generation models.
ANN indexes: Exact nearest-neighbor over millions of vectors is too slow, so vector stores use approximate indexes (HNSW being the common one) trading a little recall for orders-of-magnitude speed. 'Vector database' = this index plus filtering and CRUD.
Practical gotchas: Vectors from different embedding models are incompatible - changing models means re-embedding the whole corpus. Embed at the chunk granularity you retrieve at. Metadata filters (tenant, date, ACL) belong in the vector store query, not post-filtering.

Prompting vs RAG vs fine-tuning

The decision interviewers most want you to get right. The order of escalation is prompt -> RAG -> fine-tune, and the choice hinges on whether the gap is knowledge or behavior.

Closing the gap between the base model and your task

Prompt engineering

Instructions + few-shot examples in the request. Zero training, changes ship in minutes, works across model upgrades. Ceiling: context cost per request, and it cannot teach genuinely new knowledge or deeply consistent style.

RAG

Inject retrieved knowledge at request time. The answer when the problem is the model doesn't know our data - private, fresh, or per-tenant content - and when citations/access control matter. Adds retrieval infrastructure and its failure modes.

Fine-tuning

Further training on hundreds-to-thousands of examples. The answer when the problem is behavior, not facts: rigid output formats, house style, a narrow classify/extract task, or distilling a big model's behavior into a small cheap one. Costs data curation, training, per-model-version repetition, and it is NOT how you add knowledge - a fine-tuned model still hallucinates facts it never saw and goes stale the day after training.

When to choose each

Exhaust prompting first - it is the cheapest experiment. Add RAG when failures are knowledge-shaped ('it doesn't know X'). Fine-tune when failures are behavior-shaped after prompting plateaus ('it knows, but won't consistently do'). They compose: RAG for facts + a fine-tuned small model for form is a classic cost play.

Function / tool calling

The mechanism: You send tool definitions (name, description, JSON-schema parameters) with the request. The model does not execute anything - it replies with a structured 'call get_weather with {city: Tokyo}' message; YOUR code runs the function and sends the result back, and the model continues with it. The model is a planner emitting intents; execution, retries, and permissions stay in your code.
The agent loop: An 'agent' is this in a while-loop: model picks a tool, code executes, result appended, repeat until the model answers or a step/budget cap trips. Always cap iterations and spend - a confused model will happily loop.
Design rules: Descriptions are prompts - the model chooses tools by reading them, so write them like docs for a new hire (what it does, when to use it, when NOT to). Few, distinct tools beat many overlapping ones. Validate arguments before executing: schema conformance is not guaranteed.
Safety line: Treat model-initiated calls as untrusted input to your execution layer: allowlist the tools, scope credentials per tool, require human confirmation for destructive actions. Tool results fed back are also an injection surface (a scraped webpage can contain 'ignore your instructions').

Eval basics

The difference between a demo and a product is an eval suite. 'How would you know your prompt change helped?' is now a standard interview question.

Why evals: LLMs are nondeterministic and prompts interact unpredictably - a tweak that fixes one case silently breaks three others. An eval = a dataset of real inputs + a scoring method, run on every prompt/model change like a test suite. No eval, no safe iteration.
Scoring methods: Exact/normalized match for classification and extraction; code-based checks for anything verifiable (does the JSON parse? does the SQL run? do citations exist in the source?); LLM-as-judge for open-ended quality - a strong model grades outputs against a rubric. Judges drift and self-flatter, so calibrate them against a sample of human labels.
Building the dataset: Start small and real: 20-50 cases from actual usage plus known failure modes beats 5,000 synthetic ones. Every production failure becomes a new eval case - the suite is a regression net that only grows.
Metrics that matter: Task accuracy first, then: faithfulness/groundedness for RAG (claims supported by retrieved context - this is the hallucination metric), format validity for structured output, refusal correctness for safety, and always cost + latency per case - a 2% quality win at 5x the cost is usually a loss.

Common interview questions (and the shape of a good answer)

·Why do LLMs hallucinate, and how do you reduce it? - They generate plausible continuations, not verified facts; there is no internal truth check. Mitigate: RAG with 'answer only from context', citations the UI can verify, low temperature for factual tasks, groundedness evals, and letting the model say 'I don't know'. Reduce, not eliminate.
·Temperature 0 vs top-p - explain both dials (rescale distribution vs truncate its tail), that you tune one at a time, and that temperature 0 still is not bit-identical across runs on served models.
·When fine-tune vs RAG? - Knowledge gaps -> RAG; behavior/format gaps -> fine-tune; always prompt-engineer first. Bonus points for 'fine-tuning does not add knowledge'.
·Design a support chatbot over our docs - chunk + embed docs, hybrid retrieval + rerank, system prompt with grounding + refusal rules, citations, tool call for ticket handoff, evals for faithfulness, prompt-injection posture for user input.
·What is prompt injection? - Untrusted content (user input, scraped pages, tool results) containing instructions the model follows, overriding yours. Defenses are layered and imperfect: privilege separation via system prompt, delimiting untrusted data, output validation, least-privilege tools, human-in-the-loop for consequential actions.
·Why is the model's answer cut off? - Output token cap (check finish_reason), not the context window; know the difference.
·How do you keep LLM costs sane? - Cache repeated prefixes (prompt caching), route easy traffic to small models, cap output tokens, trim conversation history, batch offline work, and monitor per-feature token spend - cost is an eval metric, not an afterthought.
·Stateless model, 'remembering' chat? - The app resends prior turns in the context window every request; long chats hit the window and need summarization or truncation strategies.

How B-tree indexes actually work, composite index column order, covering indexes, reading EXPLAIN ANALYZE, why the planner ignores your index, join algorithms, N+1, keyset pagination, and the 'why is this query slow' scenarios interviews are built on.

Practice the patterns

Reading is the floor. The signal in interviews comes from working problems out loud and defending your tradeoffs. Spin up an AI mock interview or run a coding challenge to put these to work.

Coding challenges AI mock interview