Scaling Search Engineering at DoorDash: From Monoliths to Custom Search Engines with Satish Saley

SummaryIn this episode of Overcommitted, Satish Saley, a senior software engineer with extensive experience on DoorDash's search platform team, discusses scaling search systems at a hyper-growth company. This conversation dives deep into software engineering challenges and software development strategies that impact programmer productivity and career growth. Satish details two major engineering transformations: rebuilding the search indexing pipeline with Kafka, Flink, and Elasticsearch, and later replacing Elasticsearch with a custom search engine based on Apache Lucene, which significantly improved performance and reduced costs.The episode also explores the complexities of migrating off monoliths, securing leadership buy-in for technical rebuilds, and why respecting legacy systems is crucial in engineering culture. Additionally, the hosts share personal stories of database pivots and conclude with a confessional segment on technical decisions that didn't age well. This episode is packed with insights for software engineers and tech professionals interested in the intersection of technology, scaling, and work life balance.LinksSatish Saley on LinkedIn: https://www.linkedin.com/in/satish-saley-65527525/ Conway's Law: https://en.wikipedia.org/wiki/Conway%27s_law Build Faster Indexing with Apache Kafka and Elasticsearch: https://careersatdoordash.com/blog/open-source-search-indexing/ Introducing DoorDash's In-House Search Engine: https://doordash.engineering/2024/02/27/introducing-doordashs-in-house-search-engine/HostsOvercommitted: https://overcommitted.devBethany Janos: https://github.com/bethanyj28Brittany Ellich: https://brittanyellich.com