How Slack Rebuilt Its Search Infrastructure for Scale

Slack's search once ground to a crawl as workspaces grew. In 2024, its engineering team rewrote the search layer from a custom Lucene-backed index to a real-time Elasticsearch pipeline spanning 17 data regions. Hosts Lucas and Luna walk through the architecture: why they migrated from DocIds to opaque document keys, how they handled reindexing 2PB of data without downtime, and the tradeoff between query freshness and index latency. This episode gets into the concrete decisions—like why they chose a 90-second refresh window and how they used a dual-read pattern during cutover. If you're building anything that needs to scale from 10 users to 10 million, this one's for you. #Slack #SearchInfrastructure #Elasticsearch #Lucene #DataMigration #Indexing #RealTimeSearch #EngineeringLeadership #TechnicalDebt #SystemDesign #Scalability #Infrastructure #BusinessAndTechnology #CTOPodcast #FexingoBusiness #BusinessPodcast #EngineeringOrg #Architecture Keep every episode free: buymeacoffee.com/fexingo