#45 Michal Klos, Localytics and the World of Big Data

Summary Michal Klos of Localytics tells me about their big data stack and where he thinks the industry is going. Details Who he is, what he does; overview of the world of big data, history, batch processing, stream processing and micro batching; databases, Apache Spark, separating storage and compute; where he thinks the industry is going in the next five years, more about Spark, data lakes, query federation, Presto; how to get started with a big data project, picking technologies, doing a test; most big data projects fail, you should start small, get cross team involvement; how to scale to petabytes, start small with short expected lifespan; technologies Localytics uses, blog, they are hiring.