How One Engineer Parallelized a Batch Job from 4 Hours to 3 Minutes
from The Software Engineering Podcast with Fexingo: Code, Architecture, and Engineering Best Practices Hosted by Fexingo
In this episode, Lucas and Luna explore how a single engineer at a mid-sized fintech company transformed a legacy batch-processing pipeline that ran for four hours every night into a parallelized system that finishes in under three minutes. They walk through the specific technical decisions: partitioning the input data into 80 independent shards, using a fan-out pattern with AWS Lambda and SQS, handling partial failures with a dead-letter queue and idempotency keys, and the observability setup that made debugging the parallel version possible. The conversation touches on the trade-offs between parallelism and cost, the surprising bottleneck that was a shared PostgreSQL sequence, and why the engineer chose not to use a framework like Apache Spark. This episode is a concrete case study in how to think about concurrency in real-world systems, not just theory. #BatchProcessing #Parallelism #AWS #Lambda #SQS #Fintech #Concurrency #Performance #Optimization #DeadLetterQueue #Idempotency #PostgreSQL #Observability #CloudComputing #SoftwareEngineering #Technology #FexingoBusiness #BusinessPodcast Keep every episode free: buymeacoffee.com/fexingo
Embed this episode
No transcript yet.