Design a Payments / Checkout System (Stripe-style)

Network failures are the default, not the exception. A merchant's server submits a charge, the response times out, the merchant retries. Without idempotency, the customer is charged twice. With idempotency, the second request returns the original response.

Client contract
Every mutating request includes an Idempotency-Key header (UUID). Stripe's contract: 24-hour window, replay returns the original response (including original status code), reusing a key with a different request body returns 409 Conflict.

Server-side flow

1. Hash the request body. Compute (idempotency_key, request_hash).
2. Conditional write to the idempotency store: SET key → (request_hash, status="in_progress", started_at) IF NOT EXISTS.
3. If the conditional write fails (key exists):
   • If existing.request_hash != current.request_hash → 409 Conflict.
   • If existing.status == "completed" → return existing.response.
   • If existing.status == "in_progress" → return 409 with retry-after, OR block-and-poll (trade-offs below).
4. Otherwise, process the charge. On completion, atomically update: SET key → (request_hash, status="completed", response).

The in-progress trap
If our process dies after step 2 but before step 4, the key is stuck "in_progress" forever. Two solutions:

• TTL on the in-progress state (e.g., 60 seconds). After expiry, it can be reclaimed.
• Recovery worker: scan for in-progress keys older than threshold, query the processor for actual state, finalize.

The processor query is the safety net: even if our records are wrong, we can ask Visa "did this auth succeed?" and recover ground truth.

Storage choice
DynamoDB conditional writes are perfect here. Spanner if you need cross-region linearizability. Don't use Redis - it's not durable enough for this; a Redis failure should not cause double-charges.

Key scope
Idempotency keys are scoped to (merchant_id, key) - merchants can use the same UUID without collision. Scope is critical for multi-tenant systems.

The ledger is the source of truth for money. Get this wrong and you cannot recover. The discipline is double-entry bookkeeping borrowed from accounting.

Double-entry model
Every transaction produces matched debits and credits across accounts. A $100 charge:

• DEBIT customer_account: -$100
• CREDIT merchant_account: +$97 (after fees)
• CREDIT platform_revenue: +$3 (fee)

Sum across all accounts is always zero. Balance of any account = sum of entries.

Why double-entry

1. Auditability - every cent is accounted for, with two-sided proof.
2. Reconciliation - matching against external reports is mechanical.
3. Bug detection - if balances don't sum to zero, you have a bug, and you find it immediately.

Append-only
The ledger is never updated. Refunds, reversals, adjustments are new entries that net against the original. This preserves history for auditors and regulators (mandatory under SOX, PCI, KYC/AML).

Strong consistency required
A user with $100 balance who initiates two simultaneous $80 charges should have one succeed and one fail - not both succeed leaving them at -$60. The naive read-then-write loses to the race; the correct primitive is a conditional update or a transactional ledger write with row-locking.

For Postgres: SELECT FOR UPDATE on the account row, then INSERT into ledger_entries, then UPDATE the cached balance. Wrap in a transaction.

For DynamoDB: optimistic concurrency on the account record (version number), retry on conflict.

For Spanner: just use a transaction; it handles serialization.

Why not eventually consistent?
Eventual consistency for balances means double-spends are possible during the inconsistency window. Even a 100ms window is enough to be exploited by a malicious user. The cost of strong consistency (10-50ms write latency, geographic constraints) is non-negotiable for the money-touching path. Reporting and analytics can be eventually consistent because they're read-only.

Sharding
Ledger sharded by account_id (or merchant_id). All entries for one account live on one shard - intra-account transactions are single-shard, fast. Cross-account transactions (between two merchants on the platform) need a 2-phase commit or a saga pattern - rare, but the design must accommodate.

The hardest failure mode in payments. We sent the charge to Visa. Visa charged the card. Our process crashed before recording the result. Now we have an unrecorded charge - the customer was charged, our system thinks it never happened.

Why this is unavoidable
The processor call and our local commit cannot be one atomic operation. The network sits between them. Any sufficiently long pause between "charge succeeded" and "wrote to our ledger" is a window for failure.

Mitigation 1: pre-write before processor call
Before calling the processor, write a "pending" entry to the ledger with a unique processor_request_id. After the processor responds, update to "succeeded" or "failed". On crash recovery: scan for "pending" entries older than threshold, query the processor with processor_request_id, reconcile.

Mitigation 2: idempotent processor calls
Modern processors support idempotency keys themselves. Stripe accepts an Idempotency-Key on charge requests; replaying with the same key returns the original result without double-charging. Use this. The processor's idempotency window is your safety net for in-flight retries.

Mitigation 3: reconciliation
Daily reconciliation against processor settlement reports catches anything missed by the above. A charge in the processor's report but not in our ledger triggers an ops alert and a backfill.

Mitigation 4: don't acknowledge to the merchant until ledger is durable
The order is: pre-write pending → call processor → write final state → respond to merchant. If we respond before the final write, the merchant might think the charge succeeded while our system has nothing recorded. The merchant should never see "succeeded" without our ledger reflecting it.

The 0.001% case
Even with all of the above, there's a tiny window where the processor charged the card and we never recovered the response (processor outage during reconciliation, etc.). Final fallback: every charge has a customer_visible receipt path. Customers who notice a charge that doesn't appear in the merchant system file disputes; ops investigates. This is the manual failsafe.

The interview signal: candidates who name this failure mode and discuss the layered defenses score significantly higher than those who treat the processor call as a single atomic operation.

Card data (PAN, CVV, expiry) is heavily regulated. Any system that touches it falls under PCI-DSS - quarterly audits, network segmentation, encryption everywhere, access logging, key rotation. The audit cost is enormous.

The architecture answer: minimize what touches card data
Most of the system should never see a PAN. Tokenization confines card data to a small, hardened subsystem.

Tokenization flow

1. Browser collects card data via Stripe.js / Elements (a JavaScript library hosted by the processor). The card data is sent directly from the browser to the processor; our servers never see it.
2. Browser receives an opaque token (e.g., "tok_visa_4242"). Token is meaningless without processor's keys.
3. Browser submits the token to our backend along with the order.
4. Our backend uses the token in subsequent processor calls.

This pushes ~95% of our infrastructure out of PCI scope. Only the processor's frontend SDK and our token-handling code are in scope.

Network-isolated tokenization service (if we are the processor)
If we are Stripe (the processor), we have to handle PANs ourselves. Architecture:

• Tokenization service in a separate VPC with no outbound internet, no SSH, encrypted disks, key management via HSM.
• Application services call tokenization to convert PAN → token at write, token → PAN at processor-bound calls (network call to issuing bank).
• All access to the tokenization service is logged and reviewed.
• Application services store only tokens, never PANs.

Vault encryption
Data at rest in the vault is encrypted with a per-record DEK (data encryption key) wrapped by a master KMS key. KMS is HSM-backed (AWS KMS, GCP KMS, or HashiCorp Vault with HSM). Rotating master keys requires re-wrapping DEKs; rotating DEKs requires re-encrypting records.

Why this matters at interview time
Interviewers from payments companies probe PCI awareness. Saying "we use Stripe Elements to tokenize on the client" or "we have a network-isolated vault" earns serious credibility. Saying "we encrypt the database column" reveals a candidate who hasn't worked in payments.

Webhooks are how the merchant learns about state changes (charge.succeeded, refund.completed, dispute.opened). They're an async distributed system in their own right, and a frequent source of merchant pain.

Delivery guarantee: at-least-once
We promise to deliver each event at least once. We do not promise exactly-once - that's the merchant's problem to handle via idempotent event_id processing.

Retry schedule
Standard pattern: exponential backoff with jitter. Stripe's schedule is roughly: immediate, 5 min, 30 min, 2 hr, 12 hr, 24 hr, then 24-hour intervals up to 3 days. After ~17 attempts over 3 days, mark as undeliverable; surface in the merchant dashboard.

Signing
Every webhook request includes an HMAC signature (X-Stripe-Signature header) over the payload + timestamp using a per-merchant signing secret. Merchants verify the signature to ensure the event is genuinely from us, not a forged attack on their endpoint.

Per-endpoint queuing
Each merchant endpoint has its own delivery queue. A slow or failing endpoint can't back-pressure other merchants' deliveries. Within an endpoint, events are delivered in order (per object_id ordering matters for state-machine consumers).

Failure modes the merchant will hit

1. Their endpoint is slow → our retries pile up. Mitigation: per-endpoint concurrency limit (e.g., 10 in flight); circuit-breaker if error rate spikes.
2. Their endpoint returns 5xx → retry per schedule.
3. Their endpoint returns 4xx (other than 429) → still retry; many merchants accidentally return 400 for known events.
4. Their endpoint is unreachable for 3 days → mark undeliverable. Merchant can replay missed events from the dashboard.

Backfill
Merchants can request a replay of all events for a time window. Critical for new integrations (catch up from when they signed up) and for bug recovery (their system was down; replay to refill state).

Out-of-order delivery
Despite ordered queuing, network reordering means merchants can see refund.created before charge.succeeded if they retry stuck events. Merchants must reconcile against final state, not assume in-order events.

The "charge" abstraction hides wildly different settlement timelines depending on payment method.

Cards (sync)
Authorization happens in real time (~1-2 seconds). Capture happens immediately or up to 7 days later. Settlement (money actually moves) happens 1-3 business days after capture. The merchant sees "succeeded" within seconds; the money lands days later.

ACH (async)
Authorization is essentially trust-based - we submit a debit request to the bank, and 3-5 business days later we learn whether it cleared. The "charge" succeeds initially as "pending" and only confirms days later. Reversals (NSF, returned ACH) can happen weeks later.

BNPL / installments
Customer is approved by the BNPL provider in real time (sync), but the merchant is paid in full upfront and the customer pays the BNPL provider in installments. From our system's view: sync at charge time, but the customer's actual payments happen over months.

Wallets (Apple Pay, Google Pay)
Sync from the user's perspective (face ID / fingerprint). Underlying mechanism is a tokenized card; same settlement timeline as cards.

Crypto (async, irreversible)
Customer initiates a transfer; we wait for N confirmations (10 min - 1 hour for BTC; less for ETH); then mark succeeded. Once confirmed, irreversible - no chargebacks.

Design implication: state machine per payment method
Each payment method has its own state machine. Charge.status can be pending, processing, requires_action, succeeded, failed, refunded. Different methods take different paths. The orchestrator dispatches to method-specific handlers.

Webhook implication
Async methods produce many state changes (created → pending → processing → succeeded → potentially_reversed). Each change is a webhook. Merchants need to handle the full lifecycle, not just "succeeded".

3DS / SCA (regulatory async)
European PSD2 requires Strong Customer Authentication for many card transactions. The flow: charge starts → bank requires 3DS challenge → user redirects to bank's auth page → returns to merchant → charge completes. This injects an async step into a previously-sync flow. State: requires_action → user_completed → succeeded.

Refunds are not just "reverse the charge". They have their own lifecycle, accounting, and edge cases.

Refund types

• Full refund: returns the full charge amount to the customer's card.
• Partial refund: returns a portion. Multiple partial refunds can sum to the full amount.
• Refund of fees: when supported by the processor; usually fees are not refunded to the merchant.

Refund flow

1. Merchant calls POST /charges/{id}/refunds with amount.
2. Validate: amount + sum(prior refunds) <= original charge amount.
3. Write a "pending refund" ledger entry: DEBIT merchant_account, CREDIT customer_account.
4. Call processor refund API with idempotency key.
5. On success: update ledger entry to "completed". Webhook charge.refunded.
6. On failure: update to "failed", reverse the ledger entries. Webhook refund.failed.

Settlement: the refund amount is deducted from the merchant's next payout. If the merchant's balance is insufficient, they may owe the platform - this triggers a separate collections flow.

Disputes (chargebacks)
The customer claims a charge to their bank ("I didn't authorize this", "product never arrived"). Bank notifies the network (Visa); network notifies us; we notify the merchant. Merchant has ~7 days to provide evidence (receipts, shipping confirmation, customer communication). Network adjudicates.

Outcomes:

• Merchant wins → funds returned.
• Merchant loses → customer keeps refund; merchant also pays a chargeback fee (~$15).

Accounting impact
Disputes create a hold on the merchant's funds for the dispute amount during adjudication. Lost disputes become a permanent debit. High dispute rates trigger network penalties (Visa's VAMP program) which can disable card processing entirely.

Idempotency for refunds
Same as charges - merchant retries must not double-refund. Refund API accepts an idempotency key.

The reconciliation surface
Refunds, disputes, and processor fees all show up in daily settlement reports. Reconciliation must account for all of them - any unreconciled entry is an alert. Stripe's balance_transaction object is the canonical model: every event that moves money is a balance_transaction with a known type (charge, refund, dispute, fee, payout, adjustment).