#33: Modern Erasure Codes for Distributed Storage Systems

Traditional disk based storage arrays commonly use various forms of RAID for failure protection and/or performance improvement. Modern distributed storage systems are built as shared nothing architectures using commodity hardware components and have different failure characteristics. RAID has changed very little ever since its introduction three decades ago. There are several new and powerful constructions of erasure codes in the recent past (post- 2007) that were handcrafted to meet the needs of modern storage architectures. Several notable ones are fountain codes, locally repairable codes (LRC) and regenerating codes. However, these codes have made very little impact in the industry, because of the highly resistant nature of the storage vendors to investigate this space. The subject of erasure codes is core to storage technologies, but is sadly ignored by most storage conventions. One of the reasons could be the complexity involved in understanding the space. This talk will provide a non-mathematical, non-marketing, no-nonsense spin to the space of modern erasure codes. By the end of this talk, the audience will be able to pick specific categories from a bouquet of erasure codes that best suit a given storage system design. Learning Objectives: 1) History and classification of erasure codes; 2) Technical deep-dive into modern erasure codes; 3) Storage system parameters’ trade-off; 4) Storage system design & suitable codes; 5) Who is doing what in this space.