Design a Chat System (WhatsApp / Slack)

WebSockets are the dominant choice. SSE doesn't support client-to-server messages over the same channel; long-poll has too much per-message overhead at this scale.

Per-node connection capacity
A well-tuned Linux box (jumbo NIC, tuned TCP, kernel with epoll) can handle 100K-500K idle WebSocket connections per process. Per-process memory is dominated by TCP send/receive buffers (~5KB each, default). Tune buffers down for chat (small messages → small buffers).

Sharding strategy
Don't shard by user_id - it locks a user to a single failed node. Instead: client connects to the regional load balancer; LB routes to any healthy node with capacity. Once connected, the connection registry records (user_id → node_id, connection_id) so other services can deliver to it.

Reconnection storm
If a node fails, 100K-500K clients reconnect simultaneously. Mitigation: clients use exponential backoff with jitter on reconnect. Server side: ELB/ALB limits new conns/sec per IP and per node.

Connection registry
Hot KV store (Redis Cluster). On connect: SET user_id:device_id → node_id, TTL 60s. Heartbeat every 30s refreshes TTL. On graceful disconnect: DEL. On node failure: TTL expires, the user is implicitly offline. This trades a 60s "last-seen" lag for not needing distributed consensus on connection state.

Multi-device
A user can have 4 devices (phone, tablet, laptop, web). Each device gets its own connection. Connection registry is keyed by (user_id, device_id). Delivery iterates over all devices.

Per-conversation order is strict; cross-conversation order is irrelevant.

ID scheme
Per-conversation Lamport-style counter. The message service maintains (conversation_id → next_message_id) in a fast KV with strong consistency (DynamoDB conditional update, Spanner, or Raft-coordinated). On send: atomic increment, attach ID, persist message, publish event.

Why per-conversation rather than global? Two reasons. (1) Global Snowflake IDs are time-ordered but not strictly increasing across machines under clock skew - in a chat where order matters, that's a defect. (2) Coordination is cheap when scoped to a single conversation (one row in the KV), but expensive globally.

Sender clocks vs server clocks
Don't trust the client's clock. Order by server-assigned ID, not client timestamp. Show the server timestamp on the receiver's screen.

Out-of-order delivery
Even with strict server ordering, network reordering can deliver message 5 before message 4 to a recipient. The recipient client reorders by ID before display. Client-side buffer of 1-2 seconds is usually enough to absorb reordering.

Idempotency
Client retries on send (network flake). Each retry must produce the same server-side ID, not duplicate. Solution: client includes a UUID (client_message_id) on send. The server: "if (conversation_id, client_message_id) already exists, return existing ID; else assign new". Idempotent within the conversation.

Deletion / edit semantics
Tombstone the message (mark deleted=true, content=null). Don't actually delete - downstream consumers (search index, archive) need the event. Show "this message was deleted" in the UI.

Presence ("is X online right now?") is deceptively hard at 500M concurrent users.

Naive approach
Connection registry has user_id → online. Every contact polls. Fails: a user with 500 contacts triggers 500 lookups every few seconds.

Subscription approach
Clients subscribe to presence updates for users in their contact list. Presence service publishes connect/disconnect events to a per-user pub/sub topic. Subscribers receive deltas only.

At 2B users × 500 contacts each = 1T subscriptions. Doesn't fit in any single system. Shard the presence service by the observed user (the one whose status is published), and clients open subscriptions to all shards their contacts live on. In practice: ~100 presence shards globally, each with ~5B subscriptions.

Last-seen vs online
"Online" is binary (have an active connection). "Last-seen" is a timestamp updated on disconnect. Both are derived from the connection registry.

Privacy
Most products let users hide presence. The presence service consults a privacy table before publishing. Hidden users still appear "online" to themselves and explicit allowlist (e.g., favorites).

Scale relief
Most contacts a user has are dormant. Don't subscribe to all 500 contacts on connect - subscribe lazily as the user opens conversations. This reduces active subscriptions by 10-100x.

WhatsApp's actual approach: presence is best-effort, eventually consistent, with privacy controls. Presence drops during cell tower handoffs; users are familiar with the model. Don't over-engineer.

1:1 is easy: one sender → one recipient. Groups are a fanout problem that scales with group size.

Small groups (≤100 members)
Synchronous fanout. Message service writes the message once to the message store, then publishes one delivery event per member. Members' connection nodes push to their sockets. Total work: O(group_size) per message. Trivial.

Medium groups (100 - 10K)
Async fanout via a dedicated queue. Avoid blocking the sender's send-ack on the full fanout. Sender sees "delivered to server"; recipients see message within a couple of seconds.

Large groups (10K - 200K)
Pure fanout-on-send is expensive. A 100-message-burst in a 200K group = 20M deliveries.

Strategies for large groups

1. Hierarchical fanout: split the member list into chunks (1K each); fanout workers process chunks in parallel.
2. Pull-based reads for inactive members: only fanout to currently online members. Inactive members' clients pull on next open via "messages since last_read" API.
3. Mute / read-only optimizations: members who muted notifications don't get push - skip the push notification path.
4. Read receipts compression: don't fanout per-member read receipts to the entire group. Aggregate into "X people have read" indicators.

Group membership lookup
A 200K-member group has a 200K-row membership list. Cache it in Redis keyed by group_id. Update incrementally on join/leave. Eviction policy: LRU; cold groups rebuild from the source DB.

The very large channel (Slack #general at a 50K-person company)
Treat as a fanout-on-read product. Members pull on open; only mentions trigger fanout-on-send.

When a recipient is offline (no active connection), the message must reach them via OS push (APNs / FCM).

Trigger
Connection registry says recipient is offline → message service publishes a 'push' event to the push pipeline.

Push pipeline

1. Lookup device tokens for the user (a user can have multiple devices).
2. Apply per-device preferences (DND windows, mute settings, group-level mutes).
3. Format payload (sender, message snippet, badge count).
4. Send to APNs/FCM via batched HTTP/2.
5. Track success/failure (retry transient failures; mark token invalid if APNs returns InvalidToken).

Dedup across re-arrival
A user comes online while a push was in-flight. They get both an in-app notification and an OS push. Dedup is impossible without coordination - mitigation: clients suppress the notification banner if the conversation is currently focused.

Badge count
Source of truth: count of unread messages per user. Maintained server-side. Sent on each push. Resync on app open.

End-to-end encryption interaction
With E2EE, the server can't read message content. Push payload contains only metadata (sender, conversation_id). Client decrypts the message after receiving the push via WebSocket. APNs alert text is generic ("New message"). UX is degraded but it's the price of privacy.

Failure modes
APNs/FCM down → backoff and retry; messages still sit in the message store, available on next online connection. Don't block the messaging path on push success.

100B messages/day × 200 bytes × 365 × 2y = 14.6 PB hot. Much of this is rarely read.

Hot tier (last 30 days)
Cassandra / HBase. Partition key = conversation_id, clustering key = message_id (descending). Range scan on partition for "fetch last N messages" is the dominant query. Sub-100ms.

Warm tier (30 days - 1 year)
Same store, different cluster, smaller compute. Slower reads acceptable.

Cold archive (>1 year)
S3 / Glacier / proprietary cold storage. Indexed by (conversation_id, time_range). Restored to warm on demand for full-history searches.

Per-user inbox vs per-conversation log
Two valid models.

• Per-user: each user has a list of messages they received. Easy "all my unread" query. Bad for groups (write amplification = group size per message).
• Per-conversation: messages stored once per conversation. Each user has a (conversation_id → last_read_message_id) cursor. Reads are conversation-scoped. Writes are constant.

The per-conversation model wins at scale and is what WhatsApp/Signal/Telegram use.

Read pointers
Per (user, conversation): last_read_message_id. Updated on read. Used for unread counts and read receipts. Tiny rows; updated frequently. Hot KV store, eventually consistent.

Search
Out of band. Async indexer reads message events from Kafka, writes to ElasticSearch keyed by (user_id, conversation_id, message_id). Search is per-user (privacy) and conversation-scoped. Cold-tier messages need re-indexing on first search.