Systems

Kafka & Messaging Systems Cheat Sheet

SystemsFREELast updated: July 2026 · By gitGood Editorial

Topics, partitions, and consumer groups, the three delivery semantics and how Kafka actually achieves each, ordering guarantees, rebalancing, retention vs compaction, and a straight Kafka vs SQS vs RabbitMQ vs Kinesis comparison.

How messaging shows up in interviews

Two places: the system design round, where 'put a queue between these services' is the standard decoupling move and the follow-ups probe whether you know what that queue actually guarantees, and backend screens, where 'what happens if the consumer crashes mid-message' separates people who have run Kafka from people who have read about it. The high-leverage material: what a partition buys you (ordering + parallelism), why exactly-once is really at-least-once plus idempotency, what triggers a rebalance, and when a simple queue (SQS) beats a log (Kafka). Kafka is the vocabulary interviewers default to, so the sheet leads with it, but the semantics transfer to every broker.

Kafka core concepts

The mental model: Kafka is not a queue - it is an append-only, partitioned, replicated log that consumers read at their own pace.

Topic: A named stream of records (orders, clicks). Purely logical - the physical unit is the partition. Producers append to a topic; consumers subscribe to it.
Partition: An ordered, immutable, append-only log. A topic is split into N partitions for parallelism: N partitions = at most N consumers in a group doing useful work. Ordering is guaranteed only within a partition, never across the topic.
Offset: A record's position within its partition. Consumers track progress by committing offsets - the record itself is never deleted on read (unlike a queue). Reprocessing = seek back to an older offset.
Consumer group: A set of consumers sharing a group id. Kafka assigns each partition to exactly one consumer in the group, so the group collectively reads every record once (work-sharing). Two different groups each get the full stream (fan-out). This one mechanism gives you both queue and pub/sub semantics.
Consumer lag: Latest offset minus committed offset, per partition - how far behind a consumer is. The single most-watched Kafka health metric: growing lag means consumers cannot keep up.
Broker / partition leader: A Kafka server. Each partition has one leader broker that serves all reads and writes for it, plus followers that replicate it. Losing a broker triggers leader election for its partitions.
Replication factor & ISR: Each partition is copied to RF brokers (RF=3 is standard). The ISR (in-sync replicas) is the subset that is caught up; a write is 'committed' when all ISR members have it. min.insync.replicas=2 with acks=all means a write fails rather than being acknowledged under-replicated.
Partition key: Producers hash a key (user id, order id) to pick the partition. Same key -> same partition -> ordered. No key -> round-robin/sticky, better spread but no per-entity ordering.

Delivery semantics

The classic trio, plus what each actually means in Kafka terms. Duplicates and loss are configuration outcomes, not luck.

At-most-once vs at-least-once vs exactly-once

At-most-once

Commit the offset BEFORE processing (or produce with acks=0). If the consumer crashes after committing but before finishing, the record is lost - but it is never processed twice. Cheapest, and rarely what a business flow wants.

At-least-once

Process first, commit the offset AFTER. A crash between processing and commit means the record is redelivered and processed again - never lost, sometimes duplicated. The practical default; it obligates every consumer side effect to be idempotent.

Exactly-once (effectively-once)

In Kafka: idempotent producer (dedupes broker-side retries) + transactions (atomically write output records and commit offsets together) gives exactly-once within a Kafka-to-Kafka pipeline. The moment a side effect leaves Kafka (call an API, charge a card), you are back to at-least-once + your own idempotency keys.

When to choose each

Default to at-least-once with idempotent consumers for anything that matters. At-most-once only for lossy-tolerant telemetry. Say 'effectively-once' in interviews and explain it as at-least-once plus dedup - claiming true exactly-once across arbitrary systems is a red flag.

Producer acks setting

acks=0

Fire and forget - producer does not wait for any acknowledgment. Highest throughput, silent loss on broker failure.

acks=1

Leader acknowledges once it has written locally. A crash before followers replicate loses the record. The middle setting people forget is still lossy.

acks=all

Leader waits for all in-sync replicas. Paired with min.insync.replicas >= 2 and producer retries + enable.idempotence=true, this is the durable configuration.

When to choose each

acks=all + idempotent producer for business data; acks=1 or 0 only when you can articulate why loss is acceptable (metrics, best-effort events).

Ordering guarantees

Within a partition: Total order, always. Records with the same key land in the same partition, so per-key order (all events for user 42, in order) is the guarantee you design around.
Across partitions: None. Two records in different partitions can be consumed in any interleaving. If a flow needs global order you are down to one partition - and one consumer's worth of throughput. Usually the right fix is per-entity keys, not global order.
Retries can reorder: With max.in.flight.requests > 1 and retries, a failed batch retried after a later batch succeeds arrives out of order. enable.idempotence=true fixes this (keeps ordering with in-flight up to 5) - it is the default in modern clients and worth saying out loud.
Consumer-side reordering: Processing a partition's records concurrently in a thread pool silently throws away the ordering the partition gave you. Keep per-key work serialized (or partition-per-thread) if order matters.

Rebalancing

A rebalance is Kafka redistributing a group's partitions across its consumers. It triggers whenever group membership or subscription changes: a consumer joins (deploy, scale-up), leaves cleanly, is presumed dead (missed heartbeats within session.timeout.ms, or the gap between poll() calls exceeded max.poll.interval.ms - the classic 'my slow message handler caused a rebalance storm' bug), or partitions are added to the topic. During a classic eager rebalance every consumer stops, gives up all partitions, and reassigns - a stop-the-world pause where lag builds. Cooperative (incremental) rebalancing, the modern default protocol, only moves the partitions that actually need to move, so the rest of the group keeps consuming. Static group membership (group.instance.id) lets a bounced consumer return within the session timeout and reclaim its old partitions without triggering a rebalance at all - useful for rolling deploys. Interview angle: know that redelivery after a rebalance is normal (offsets committed at intervals, not per record), which is another reason consumers must be idempotent.

Retention vs compaction

How the log gets cleaned

Time/size retention (delete)

Segments older than retention.ms (default 7 days) or beyond retention.bytes are deleted wholesale, read or not. The log is a replayable buffer of recent history - consumers must keep up within the window.

Log compaction (compact)

Kafka keeps at least the latest record per key and eventually removes older values for that key; a null-value record (tombstone) deletes the key. The topic becomes a durable, replayable snapshot of current state per key - the backbone of changelog topics and Kafka-as-source-of-truth patterns.

Both (compact,delete)

Compact per key AND expire by time - bounded changelog for keys that stop updating.

When to choose each

Retention for event streams where history is the point (clicks, orders as events). Compaction for state-shaped topics where only the latest value per key matters (user profile snapshots, connector offsets, CDC changelogs). Say the tombstone detail - it is the part people miss.

Kafka vs SQS vs RabbitMQ vs Kinesis

The 'which one and why' question. Frame the answer around log vs queue, ops burden, and fan-out/replay needs.

Choosing a messaging backbone

Kafka

Partitioned replicated log. Strengths: huge throughput, per-key ordering, replay from any offset, cheap fan-out to many consumer groups, long retention, rich ecosystem (Connect, Streams). Costs: operational weight (brokers/KRaft, partitions, rebalances) unless managed, no per-message delay/DLQ primitives out of the box - consumers build their own retry topics.

SQS

Fully-managed simple queue. Strengths: zero ops, effectively unlimited elastic throughput, per-message visibility timeout, built-in DLQs, pay-per-use. Costs: no replay (consumed = gone), no fan-out from one queue (pair with SNS), standard queues are unordered + at-least-once; FIFO queues give ordering per message-group at lower throughput.

RabbitMQ

Traditional broker with smart routing (exchanges: direct/topic/fanout/headers), per-message acks, priorities, TTLs, and DLXs. Strengths: flexible routing and low-latency task/RPC patterns. Costs: throughput and horizontal scaling well below Kafka; classic queues are not a replayable log (streams feature narrows this); you run it.

Kinesis

AWS's managed Kafka-shaped log: shards ~ partitions, ordered per shard, replayable within retention (up to 365 days). Strengths: no brokers to run, tight AWS integration (Lambda, Firehose). Costs: per-shard throughput caps (1 MB/s in, 2 MB/s out) and read-consumer limits unless you pay for enhanced fan-out; shard resharding is manual-ish.

When to choose each

Event streaming, replay, multiple independent consumers, or >tens of MB/s: Kafka (or Kinesis if you are all-in on AWS and want zero ops). Simple decoupled work queues with retries and DLQs: SQS. Complex routing, task queues, per-message priorities on modest scale: RabbitMQ. If the requirement is just 'buffer these jobs', say SQS - reaching for Kafka by default is a seniority tell in the wrong direction.

Gotchas and interview traps

The follow-ups that check for real operational experience.

·Partition count is a one-way door in practice - you can add partitions but never remove them, and adding them breaks key -> partition mapping (same key now hashes elsewhere), which silently breaks per-key ordering across the change.
·A hot key skews a partition - if one tenant produces 50% of traffic, its partition (and its consumer) becomes the bottleneck no matter how many partitions exist. Mitigate with key salting or tenant isolation.
·More consumers than partitions = idle consumers. Scaling the group past the partition count does nothing; plan partition count from target consumer parallelism.
·Committing offsets on a schedule means crash-redelivery is normal - anything between the last commit and the crash is reprocessed. Idempotency is not optional.
·max.poll.interval.ms vs slow handlers - a handler that takes longer than the poll interval gets the consumer evicted, its partitions reassigned, and the batch redelivered: the classic rebalance/duplicate loop. Fix by processing async with pause/resume, or raising the interval deliberately.
·The dual-write problem - writing the DB and then producing to Kafka (or vice versa) can succeed on one side and fail on the other. The fix is the outbox pattern or CDC, not a try/catch.
·Unbounded retention is not free - 'we keep everything forever' means storage math and compaction strategy, and consumers that replay from offset 0 will hammer downstream systems.
·'Exactly-once' claims - Kafka's EOS covers Kafka-in/Kafka-out pipelines. Any external side effect needs its own idempotency; interviewers deliberately poke this.

How B-tree indexes actually work, composite index column order, covering indexes, reading EXPLAIN ANALYZE, why the planner ignores your index, join algorithms, N+1, keyset pagination, and the 'why is this query slow' scenarios interviews are built on.

Practice the patterns

Reading is the floor. The signal in interviews comes from working problems out loud and defending your tradeoffs. Spin up an AI mock interview or run a coding challenge to put these to work.

Coding challenges AI mock interview

Choosing a messaging backbone

Kafka

SQS

RabbitMQ

Kinesis

When to choose each