System Design Interview: Payment Settlement Batch Processing

Design a batch processing system for end-of-day payment settlement at a payments company that processes 50 million transactions per day. The system must net merchant positions, calculate fees, and initiate fund transfers to merchant bank accounts within a strict bank cutoff window. Walk me through your design, covering reliability, scalability, and how you'd handle failures.Key TakeawaysThe outbox pattern is the canonical solution to the dual-write problem. Know it cold.Partition keys define ordering and parallelism — choose with intention, usually around the natural aggregation boundary (here, merchant ID).Throttling must be distributed when you have multiple workers — Redis-backed buckets or a sidecar.Idempotency is non-negotiable in payments. Design for at-least-once and dedupe.Retries are tiered: in-process for transient, delay-queue for slower-resolving, DLQ for terminal.Backpressure beats dropping — use Kafka lag as your buffer when downstream is slow.Reconciliation closes the loop — you don't know it worked until ground truth confirms.Corrections are new events — never rewrite history in a financial system.