Big Data and the Role of Statistical Scalability

The ability to collect and store data has increased exponentially in recent years. So too have the challenges around managing the huge volumes generated and trying to extract meaningful information from it. However, it’s universally acknowledged that such ‘Big Data’ has the potential to transform many aspects of people's lives, particularly in data-rich areas – including industries, government agencies, science and technology. Examples range from the use of the Oyster card to improve London’s transport network, to the Square Kilometre Array astrophysics project that has the potential to transform understanding of the universe.The important role of statistics within Big Data has been clear for some time. Statistical techniques, such as sampling populations, confounders, multiple testing, bias, overfitting and generally dealing with variation in the data, are essential for modelling and effective analysis. A major challenge of working with Big Data is that the volume can exceed what is