Big Data and the Role of Statistical Scalability

The ability to collect and store data has increased exponentially in recent years. So too have the challenges around managing the huge volumes generated and trying to extract meaningful information from it. However, it’s universally acknowledged that such ‘Big Data’ has the potential to transform many aspects of people's lives, particularly in data-rich areas – including industries, government agencies, science and technology. Examples range from the use of the Oyster card to improve London’s transport network, to the Square Kilometre Array astrophysics project that has the potential to transform understanding of the universe.

The important role of statistics within Big Data has been clear for some time. Statistical techniques, such as sampling populations, confounders, multiple testing, bias, overfitting and generally dealing with variation in the data, are essential for modelling and effective analysis. A major challenge of working with Big Data is that the volume can exceed what is feasible to compute with and traditional methods can fail to scale up. There has also been a tendency to focus purely on algorithmic scalability, e.g., developing versions of existing statistical algorithms that scale better with the amount of data. However, such approaches ignore the fact that fundamentally new issues often arise, and highly innovative solutions are required.

This workshop was part of the six month programme at the Isaac Newton Institute (INI) on Statistical Scalability. The Programme aimed to help address some of these issues by simultaneous consideration of the methodological, theoretical and computational challenges involved and the development of robust, scalable methods, crucial to unlocking the potential of Big Data. This event was also in collaboration with the EPSRC funded StatScale Network.

Aims and Objectives This knowledge exchange event by the Turing Gateway to Mathematics sought to extend the reach of the research being undertaken as part of the INI Statistical Scalability Research Programme. It opened up the discussion to a wide audience, including those working in multiple industrial sectors, Government and the public sector.

Because interest in Big Data is so intense, the field is developing very rapidly. This event therefore facilitated the dissemination of state-of-the-art statistical research and highlighted a number of key future research directions, such as:

Statistical inference after model selection Model misspecification Trade-offs between statistical and computational efficiency Sequential decision problems New data types There was also three end-user sessions which featured speakers from the health, energy and communications sectors. Speakers described how Big Data scaling is managed in their organisations and the challenges they face. Each session included time for discussion and feedback from the audience.

The workshop also included a poster exhibition, which ran during the lunch and the drinks/networking session and there was a short discussion and question session to finish. It was expected to bring together industrial and academic experts from a diverse set of backgrounds and areas, including healthcare, medicine, manufacturing, finance, defence, engineering, security, communications, Government and the public sector.