Streaming Ecosystem Complexities and Cost Management // Rohit Agrawal // #302

Streaming Ecosystem Complexities and Cost Management // MLOps Podcast #302 with Rohit Agrawal, Director of Engineering at Tecton.Join the Community: https://go.mlops.community/YTJoinIn Get the newsletter: https://go.mlops.community/YTNewsletter // AbstractDemetrios talks with Rohit Agrawal, Director of Engineering at Tecton, about the challenges and future of streaming data in ML. Rohit shares his path at Tecton and insights on managing real-time and batch systems. They cover tool fragmentation (Kafka, Flink, etc.), infrastructure costs, managed services, and trends like using S3 for storage and Iceberg as the GitHub for data. The episode wraps with thoughts on BYOC solutions and evolving data architectures.// BioRohit Agrawal is an Engineering Manager at Tecton, leading the Real-Time Execution team. Before Tecton, Rohit was a Lead Software Engineer at Salesforce, where he focused on transaction processing and storage in OLTP relational databases. He holds a Master’s Degree in Computer Systems from Carnegie Mellon University and a Bachelor’s Degree in Electrical Engineering from the Biria Institute of Technology and Science in Pilani, India.// Related Links~~~~~~~~ ✌️Connect With Us ✌️ ~~~~~~~Catch all episodes, blogs, newsletters, and more: https://go.mlops.community/TYExploreJoin our Slack community [https://go.mlops.community/slack]Follow us on X/Twitter [@mlopscommunity](https://x.com/mlopscommunity) or [LinkedIn](https://go.mlops.community/linkedin)] Sign up for the next meetup: [https://go.mlops.community/register]MLOps Swag/Merch: [https://shop.mlops.community/]Connect with Demetrios on LinkedIn: /dpbrinkmConnect with Rohit on LinkedIn: /agrawalrohit10Timestamps:[00:00] Rohit's preferred coffee[00:34] Takeaways[01:25] Data Streaming[06:50] Optimizing for Speed[09:17] DuckDB vs Spark Flink[13:09] Optimizing Feature Engineering Best Practices[16:42] Checkpointing Frequency Guide[23:08] Streaming Tips and Tricks[30:00] Cloud Costs vs Human Costs[32:09] Race to Learn[39:24] Right-Sized Engineering Practices[42:06] Streaming Simplicity vs Complexity[47:02] Wrap up