Skip to content
How One Engineer Parallelized a Batch Job from 4 Hours to 3 Minutes episode artwork

Jul 15, 2026 11 min

How One Engineer Parallelized a Batch Job from 4 Hours to 3 Minutes

from The Software Engineering Podcast with Fexingo: Code, Architecture, and Engineering Best Practices Hosted by Fexingo

In this episode, Lucas and Luna explore how a single engineer at a mid-sized fintech company transformed a legacy batch-processing pipeline that ran for four hours every night into a parallelized system that finishes in under three minutes. They walk through the specific technical decisions: partitioning the input data into 80 independent shards, using a fan-out pattern with AWS Lambda and SQS, handling partial failures with a dead-letter queue and idempotency keys, and the observability setup that made debugging the parallel version possible. The conversation touches on the trade-offs between parallelism and cost, the surprising bottleneck that was a shared PostgreSQL sequence, and why the engineer chose not to use a framework like Apache Spark. This episode is a concrete case study in how to think about concurrency in real-world systems, not just theory. #BatchProcessing #Parallelism #AWS #Lambda #SQS #Fintech #Concurrency #Performance #Optimization #DeadLetterQueue #Idempotency #PostgreSQL #Observability #CloudComputing #SoftwareEngineering #Technology #FexingoBusiness #BusinessPodcast Keep every episode free: buymeacoffee.com/fexingo

Episode details come from the podcast's RSS feed.

Embed this episode

Ready to play

How One Engineer Parallelized a Batch Job from 4 Hours to 3 Minutes

0:00 11:16

No transcript yet.

Requests help us decide what to transcribe next.

Frequently asked questions

How long is this episode of The Software Engineering Podcast with Fexingo: Code, Architecture, and Engineering Best Practices?

This episode is 11 minutes long.

When was this The Software Engineering Podcast with Fexingo: Code, Architecture, and Engineering Best Practices episode published?

This episode was published on July 15, 2026.

Can I download this The Software Engineering Podcast with Fexingo: Code, Architecture, and Engineering Best Practices episode?

Yes. Use the download control on the episode player to save the publisher-provided media file.