Streaming: where and when does it make sense vs batch integration; CDC best practices

What this episode covers

Real-time data is no longer a future problem. At Small Data SF by MotherDuck, I sat down with David Yaffe, Co-Founder & CEO at Estuary, to talk about what has changed in the world of data streaming!!!!A few years ago, real-time data was something most teams put on their “later” list. Expensive. Hard to scale. Too complex for most use cases.But as David shared, that story has shifted fast.Here are some takeaways from our conversation:- Streaming is now viable for everyoneWith cheaper compute, mature tooling, and simpler developer experiences, real-time data isn’t a luxury anymore. The barriers that once made it a niche capability are gone- Batch vs Real-time: Asking the right questionsBefore jumping to streaming, David suggests asking what problems you’re solving — speed for the sake of speed rarely pays off. Sometimes batch is just fine. The goal is fit, not flash- Architecture mattersMoving from batch to streaming means thinking end-to-end: from schema evolution and error handling to observability. Teams that skip this planning end up redoing pipelines- CDC done rightChange Data Capture is powerful, but it’s easy to misuse. The most common mistake? Treating CDC as an ETL replacement rather than an event system. Understanding that difference prevents pain later- The conversation was practical, focused, and refreshing.Real-time isn’t about chasing trends, it’s about enabling faster insights and cleaner data movement with less friction.If you’ve been wondering when “real-time” becomes realistic, this one will give you a clear answer.#data #ai #motherduck #smalldatasf #theravitshow

PodParley-generated summary based on available episode metadata and transcript content.