Streaming: where and when does it make sense vs batch integration; CDC best practices

Real-time data is no longer a future problem. At Small Data SF by MotherDuck, I sat down with David Yaffe, Co-Founder & CEO at Estuary, to talk about what has changed in the world of data streaming!!!! A few years ago, real-time data was something most teams put on their “later” list. Expensive. Hard to scale. Too complex for most use cases. But as David shared, that story has shifted fast. Here are some takeaways from our conversation: - Streaming is now viable for everyone With cheaper compute, mature tooling, and simpler developer experiences, real-time data isn’t a luxury anymore. The barriers that once made it a niche capability are gone - Batch vs Real-time: Asking the right questions Before jumping to streaming, David suggests asking what problems you’re solving — speed for the sake of speed rarely pays off. Sometimes batch is just fine. The goal is fit, not flash - Architecture matters Moving from batch to streaming means thinking end-to-end: from schema evolution and error handling to observability. Teams that skip this planning end up redoing pipelines - CDC done right Change Data Capture is powerful, but it’s easy to misuse. The most common mistake? Treating CDC as an ETL replacement rather than an event system. Understanding that difference prevents pain later - The conversation was practical, focused, and refreshing. Real-time isn’t about chasing trends, it’s about enabling faster insights and cleaner data movement with less friction. If you’ve been wondering when “real-time” becomes realistic, this one will give you a clear answer. #data #ai #motherduck #smalldatasf #theravitshow