Accelerated Computing in Modern Data Centers With Datapelago

Summary In this episode of the Data Engineering Podcast Rajan Goyal, CEO and co-founder of Datapelago, talks about improving efficiencies in data processing by reimagining system architecture. Rajan explains the shift from hyperconverged to disaggregated and composable infrastructure, highlighting the importance of accelerated computing in modern data centers. He discusses the evolution from proprietary to open, composable stacks, emphasizing the role of open table formats and the need for a universal data processing engine, and outlines Datapelago's strategy to leverage existing frameworks like Spark and Trino while providing accelerated computing benefits. Announcements Hello and welcome to the Data Engineering Podcast, the show about modern data management Data migrations are brutal. They drag on for months—sometimes years—burning through resources and crushing team morale. Datafold's AI-powered Migration Agent changes all that. Their unique combination of AI code translation and automated data validation has helped companies complete migrations up to 10 times faster than manual approaches. And they're so confident in their solution, they'll actually guarantee your timeline in writing. Ready to turn your year-long migration into weeks? Visit dataengineeringpodcast.com/datafold today for the details. Your host is Tobias Macey and today I'm interviewing Rajan Goyal about how to drastically improve efficiencies in data processing by re-imagining the system architecture Interview Introduction How did you get involved in the area of data management? Can you start by outlining the main factors that contribute to performance challenges in data lake environments? The different components of open data processing systems have evolved from different starting points with different objectives. In your experience, how has that un-planned and un-synchronized evolution of the ecosystem hindered the capabilities and adoption of open technologies? The introduction of a new cross-cutting capability (e.g. Iceberg) has typically taken a substantial amount of time to gain support across different engines and ecosystems. What do you see as the point of highest leverage to improve the capabilities of the entire stack with the least amount of co-ordination? What was the motivating insight that led you to invest in the technology that powers Datapelago? Can you describe the system design of Datapelago and how it integrates with existing data engines? The growth in the generation and application of unstructured data is a notable shift in the work being done by data teams. What are the areas of overlap in the fundamental nature of data (whether structured, semi-structured, or unstructured) that you are able to exploit to bridge the processing gap? What are the most interesting, innovative, or unexpected ways that you have seen Datapelago used? What are the most interesting, unexpected, or challenging lessons that you have learned while working on Datapelago? When is Datapelago the wrong choice? What do you have planned for the future of Datapelago? Contact Info LinkedIn Parting Question From your perspective, what is the biggest gap in the tooling or technology for data management today? Links Datapelago MIPS Architecture ARM Architecture AWS Nitro Mellanox Nvidia Von Neumann Architecture TPU == Tensor Processing Unit FPGA == Field-Programmable Gate Array Spark Trino Iceberg Podcast Episode Delta Lake Podcast Episode Hudi Podcast Episode Apache Gluten Intermediate Representation Turing Completeness LLVM Amdahl's Law LSTM == Long Short-Term Memory The intro and outro music is from The Hug by The Freak Fandango Orchestra / CC BY-SA