Design a Notifications / Pub-Sub System (Kafka / SNS / SQS)

The core architectural decision. It mirrors the news-feed problem but with sharper trade-offs because notifications usually require push (not pull).

Fanout-on-write (push model)
When an event arrives, immediately compute the recipient list and produce a delivery task per recipient. Writes scale with subscriber count.

Pros: low delivery latency (no waiting for the user to poll). Push notifications work (user doesn't need to be online).
Cons: write amplification = avg subscribers per event. A broadcast event (1M subscribers) = 1M deliveries to compute and queue.

Fanout-on-read (pull model)
Store events centrally; each consumer queries on connect for "events I missed since last poll". Writes are constant; reads are expensive.

Pros: minimal write amplification. Cleanly handles late subscribers.
Cons: requires polling. Latency = polling interval. Doesn't support push.

Hybrid (the production answer)

• For low-cardinality fanout (1:1, 1:few): fanout-on-write. Cheap.
• For high-cardinality fanout (broadcast to 1M users): fanout-on-write to a shared in-memory buffer per gateway; gateway pushes to all connected users (one shared event, per-user filter at gateway).
• For very-high-cardinality (broadcast to all 100M users): fanout-on-write to a "broadcast" topic; gateways subscribe to broadcasts and filter at the edge.
• Email digests (low-frequency, batched): fanout-on-read. Hourly job pulls events the user subscribed to, packages a digest.

The broadcast optimization
Naive: 1 broadcast event × 100M users = 100M delivery tasks. Wasted work.
Smart: write the event to a "broadcasts" queue. All gateway nodes subscribe. Each gateway iterates its connected users (~10K each), filters per-user preferences, pushes. Total work: 1 read per gateway × 5K gateways = 5K reads instead of 100M.

Late subscribers
A user comes online after the broadcast was sent. They missed it. Solutions:

• Backfill: on connect, gateway checks "did this user miss any in-app notifications?" via fanout-on-read against the durable log.
• Push fallback: if the user is offline at fanout time, also enqueue an OS push (APNs/FCM). When they open the app, in-app notifications are reconciled.

The interview signal: candidates who name both models and explain when each applies score higher than candidates who default to one.

Distributed messaging promises three flavors of delivery. Only one is practical at scale.

At-most-once
The producer sends; if anything goes wrong, the message is lost. Used by UDP-style protocols where loss is acceptable (logging, metrics). Not suitable for notifications - users miss real events.

At-least-once (the standard)
Every message is delivered one or more times. Duplicates happen on retries (network failures, broker restarts, consumer crashes after processing but before commit). Consumers must be idempotent on event_id.

This is what Kafka, SQS, RabbitMQ, NATS, and every other production messaging system actually delivers. Anyone who claims "exactly-once" without significant qualification is selling something.

Exactly-once (almost a marketing claim)
The message is delivered exactly once. Mathematically requires distributed transactions across producer, broker, and consumer.

Kafka has "exactly-once semantics" (EOS) via transactional producer + idempotent consumer + transactional commit of offsets. It works only when:

• Both producer and consumer are within the Kafka ecosystem.
• The "side effect" of the consumer is also a Kafka write.

The moment your consumer's side effect is "send an email" or "POST to a webhook", exactly-once is back to at-least-once + idempotency on the consumer.

Idempotency in practice
Each event has a globally unique event_id (UUID). Consumers maintain a recent-events bloom filter or KV store of "events I've already processed". Duplicates are detected and skipped.

Idempotency window: how far back to deduplicate. 24 hours is typical (matches the worst-case retry tail). Storage: ~100M events/day × 16 bytes (UUID hash) = 1.6 GB. Trivial.

Where duplicates come from

• Retries on transient failures (the message was processed; the ack failed; producer retries).
• Consumer crash after processing but before committing offset.
• Broker rebalance during processing.
• Multiple consumer instances accidentally processing the same partition.

The real-world contract
Kafka guarantees: "at-least-once delivery within retention; ordered within a partition; durable across N replicas". That's the truth. Build idempotent consumers; live happily.

Each channel has its own protocol, its own provider, its own rate limits, and its own failure modes. The architecture must isolate them.

In-app (WebSocket gateway)
Persistent connection to the user's device. On delivery: lookup connection, write to socket. If user has multiple devices, deliver to all. If no active connection, mark for push fallback.

Latency: sub-second when connected.
Failure mode: disconnected user. Fallback: enqueue push.

Push (APNs / FCM)
HTTP/2 connection to APNs (iOS) or FCM (Android). Per-device tokens stored in the user's device registry.

Latency: typically <5 seconds globally; APNs is fast.
Failure modes:

• InvalidToken (user uninstalled the app): mark token invalid; stop sending.
• Device offline: APNs holds the message for the user (last-only mode for badge/silent; first-only for visible).
• Quota exceeded: APNs allows ~9K-21K notifications per second per HTTP/2 connection. Open more connections.

Email (SES, SendGrid, Mailgun)
SMTP submission to the provider. Per-recipient.

Latency: 5-60 seconds typically; can be longer if the recipient's mailbox is slow.
Failure modes:

• Hard bounce (invalid address): mark unsubscribed; stop sending. Affects sender reputation if not handled.
• Soft bounce (full mailbox, temporary): retry over hours.
• Spam classification: degrades sender reputation; reduces deliverability for everyone.
• Rate limits: SES starts at 14/sec; warmup to 50K+/sec over weeks. Plan ahead.

SMS (Twilio, AWS SNS)
HTTP API to provider. Per-recipient. Most expensive channel.

Latency: 5-30 seconds typically.
Failure modes:

• Invalid number: mark invalid.
• Carrier filtering (spam-suspected SMS dropped): hard to detect; affects deliverability silently.
• Rate limits: long-code (regular phone number) ~1/sec; short-code ~100/sec; toll-free higher. Plan based on volume.
• Compliance: SMS is heavily regulated (TCPA in US, similar elsewhere). Opt-in must be explicit; STOP/HELP keywords are mandatory.

Webhook (HTTP POST to merchant endpoint)
For B2B notifications (Stripe-style). HMAC-signed payload.

Latency: depends on merchant endpoint.
Failure modes:

• Endpoint down: retry per schedule (1m, 5m, 30m, ..., 24h).
• Endpoint slow: per-endpoint concurrency limit; circuit breaker.
• Endpoint returns 4xx: retry but flag for ops review (likely merchant misconfiguration).

Per-channel queues
Each channel has its own queue and worker pool. A Twilio outage doesn't back-pressure email delivery. A spike in webhook deliveries doesn't crowd out push.

Channel selection logic
Per-user preferences determine which channels deliver which event types. "Mention" → in-app + push. "Daily digest" → email only. "Critical alert" → all channels. Selection happens in the fan-out stage, before queueing.

When a downstream channel degrades, the messaging substrate must absorb the failure without losing data or cascading to producers.

The failure scenario
Twilio's API starts rate-limiting us (or returns 5xx). Our SMS workers keep retrying; the queue grows; eventually it overflows.

Back-pressure at the queue layer
Per-channel queues have a configurable max depth (e.g., 1M messages). When approaching the limit:

• Workers prioritize older messages (FIFO with TTL).
• Producers (the fan-out service) slow down or stop producing new SMS deliveries.
• Critically: the durable event log (Kafka) keeps accepting events. Other channels are unaffected.

The event log is the buffer that absorbs the back-pressure. As long as Kafka has retention, no events are lost - SMS deliveries are simply delayed.

Dead-letter queue (DLQ)
A delivery that fails after N retries (e.g., 7 attempts over 24 hours) goes to the DLQ. The message is preserved with the failure reason; it's not lost.

DLQ workflow:

• Alarms fire when DLQ depth exceeds threshold.
• Ops investigate (provider issue? bug? bad data?).
• Fix the root cause.
• Replay from DLQ back to the channel queue.

The DLQ is the safety net. Without it, transient failures permanently lose messages.

Selective shedding
At extreme load, the platform may need to drop low-priority deliveries to preserve high-priority ones. Examples:

• Marketing emails dropped before transactional ones.
• "FYI" notifications dropped before "critical" ones.
• Free-tier customers' notifications dropped before paid-tier.

Shedding policies live in the fan-out service. They consult per-event priority and current queue depths.

Circuit breakers
Each channel worker pool has a circuit breaker. If the provider's error rate exceeds threshold (e.g., 50% errors in 1 min), open the breaker - stop sending for a cool-off period. Re-test with a small probe; close the breaker when healthy.

This prevents pile-on (sending more requests when the provider is already struggling) and allows the provider to recover.

Cascading failure prevention
The architecture explicitly does NOT couple producer success to delivery success. Producers see 200 the moment the event is durably logged in Kafka. If every downstream channel is down, producers are unaffected.

This decoupling is non-negotiable. Coupling producer to delivery means a Twilio outage takes down our event ingestion.

Users want control over what they receive. The platform must enforce preferences before delivery, not after - sending and then asking forgiveness damages the brand and risks regulatory exposure.

Preference model
Per (user, topic, channel): allow / deny.

user_42:
  topic="security_alert", channel="push": allow
  topic="security_alert", channel="email": allow
  topic="marketing", channel="email": allow
  topic="marketing", channel="push": deny
  topic="social_mention", channel="*": allow

Stored in DynamoDB / Redis. Looked up in the fan-out path.

Default preferences
New users get sensible defaults: transactional notifications all-channels, marketing email-only, social in-app-only. Users opt out of categories; rarely opt in to ones they don't get by default.

Quiet hours
"Don't send push notifications between 10pm-7am in the user's timezone". Implemented by checking the user's local time at delivery; defer to the next open window.

Edge case: a "critical" event (security alert) overrides quiet hours. The product team decides which categories are override-eligible.

Frequency caps
"No more than 3 marketing emails per week". Implemented by maintaining a per-user, per-category counter with a sliding window. At fan-out: check counter; if over cap, drop or queue for next window.

This is a mini rate-limiter built into the notifications path.

Bundling / digest mode
Some users prefer "send me a summary at 5pm" instead of real-time. The fan-out service writes events to a per-user pending-bundle table. A scheduled job (cron at 5pm per timezone) compiles the bundle and sends one email.

Compliance hooks

• Email: every email includes a one-click unsubscribe link (CAN-SPAM Section 5(a)(5)). Unsubscribe must take effect within 10 business days; in practice, immediately.
• SMS: STOP keyword unsubscribes; HELP keyword returns identification. Carrier-enforced.
• GDPR: per-user consent records with timestamp + source. Audit trail for consent withdrawal.

The preference service owns this state and is authoritative. Other services consult it; never bypass.

Subscription management UI
Users access /settings/notifications. Listed by topic; each topic has a per-channel toggle. Saved settings flow to the preference service via API; immediately effective on next event.

Enterprise / admin overrides
For B2B products, administrators may force certain notifications regardless of user preferences (security alerts, billing notices). The preference service has a "mandatory" flag per (org, topic) that overrides user toggles for org-internal events.

Kafka's central trick: partitions provide parallelism for free, but ordering is preserved only within a partition. The choice of partition key shapes the entire system.

Partition by what?

• By user_id: all events for one user land on one partition. Per-user ordering preserved (a 'follow' event before a 'message' event is processed in order). Cost: hot users create hot partitions.
• By event_type: skewed - a high-volume event type (page_view) overwhelms its partition.
• By tenant_id: per-tenant ordering; balanced if tenant size is uniform.
• By random hash: perfect load balancing; no ordering guarantees.

For notifications, partition by user_id is the standard. Per-user ordering matters (notifications about a single conversation should arrive in order); cross-user ordering doesn't.

Hot partition mitigation
A celebrity user (millions of incoming notifications) creates a hot partition. Mitigation:

• Sub-partition celebrity users (user_id || sub_partition) for parallelism within the user, accept slight ordering loss.
• Promote celebrity events to a dedicated topic with more partitions.

Consumer parallelism
A topic with 100 partitions can be consumed by up to 100 parallel consumers. Adding more consumers doesn't help (idle consumers). Setting partition count requires forecasting peak parallelism needed; it's painful to change after the fact.

Rule of thumb: 2-3x more partitions than expected peak consumer count. A topic expected to peak at 50 consumers gets 100-150 partitions.

Consumer groups and offset management
Multiple workers form a consumer group; Kafka assigns partitions to workers within the group. Each worker tracks its committed offset.

Failure recovery: a worker dies; Kafka reassigns its partitions to surviving workers; they resume from the last committed offset. Some duplicates possible (events processed but not committed) - hence at-least-once + idempotency.

Rebalance pauses
When the consumer group changes (worker added/removed), Kafka rebalances - all consumers pause briefly while reassignment happens. Modern Kafka (cooperative rebalancing) reduces pause to single-partition impact instead of group-wide.

Lag monitoring
The KPI for any pub-sub system is "consumer lag" - how far behind real-time the consumer is. Alert at >1 minute lag for in-app channels, >10 minutes for email.

Lag is the integral over time of (production rate - consumption rate). When production exceeds consumption, lag grows linearly. Alarm immediately - by the time you notice manually, you're hours behind.

Consumer scaling
Lag growing → add consumers (up to partition count). If at partition count, the only options are:

• Horizontal: more partitions (requires reindexing; not zero-downtime).
• Vertical: faster consumer instances.
• Reduce work per event: simplify processing, push expensive work async.