Understanding Multi-Modal Data for Social and Human Behaviour

Deciphering sequential data sets using Rough Path Theory

Background In an era of data deluge - sensors, cameras, computers and smart phones capture and store an unending torrent of data about human activity. The data is high-dimensional, sequential, complex, heterogeneous and multimodal in nature; but the sample size is woefully small in comparison.

New scientific and technological methods are emerging that can sometimes reduce this surfeit of dimension to allow meaningful and useful information to be extracted from data arising from human behaviour; by allowing patterns to be predicted for the first time there are new opportunities for significant societal benefit. These torrents of multi-modal streamed data carry vital information in so many areas of human existence. This presents huge opportunities and it is a central goal of statistics and modern computer science to extract meaning and insight in an actionable form from such data.

The field of Rough Paths theory (RPT) is a focus of the four month Research Programme at the Isaac Newton Institute (INI) on Scaling Limits, Rough Paths, Quantum Field Theory. RPT is emerging as a useful data science tool. It has at its core, the ability to describe complex behaviour concisely. RPT is focused on describing evolving systems, and crucially exploits the mathematician’s ability to describe continuous sequential information in terms of the order of events without introducing a parameterisation. This is in essential contrast to conventional ways of describing sequential data, and results in a massive and controlled dimension reduction. The result is a set of simplified descriptions of the stream that are of fixed dimension regardless of the complexity of the path, and often prove extremely well adapted to the application of modern data science techniques, allowing greater sensitivity and better learning.

RP theory has great potential as a mathematical tool to facilitate the use of data science to understand social and human data. It already has a recognised role (along with techniques like deep learning), in the recognition of Chinese handwriting on a mobile device, with billions of complex figure gestures being successfully translated into typographical Chinese characters. It has also been used successfully to differentiate individual diagnoses for mental health conditions such as bipolar disorder using simple non-invasive ‘mood zoom’ diaries from mobile phones and remarkable results have been obtained in several existing data science challenges, including the interpretation of human movement from landmark data extracted from video, etc.

Aims and Objectives Real benefits to many areas of modern society arise if one can analyse, model and predict different aspects of social and human behaviours. Techniques, such as those offered by RPT, increase the range of potential successes to include recognizing human actions and understanding changing facial expressions.

This workshop aimed to increase awareness of what is possible, whether it be better mitigation of risks, management of outcomes, or supporting individuals in their daily lives, across the spectrum of social and human behaviour.

The programme for the day featured state-of-the-art surveys, as well as several shorter presentations on success stories; together these were intended to help end-users to visualise and articulate their own data challenges in this area.

The day also included end-user talks from the security, safety and human health and behaviour areas. It was of interest to a wide range of stakeholders and specifically those who wished to gain greater insight from complex sequential data. The challenges presented by such data occur across the spectrum and so was relevant to multiple application areas and sectors including areas of engineering, security, communications, human health and social sciences areas.