Some Big Data and High Performance Data Analytics Challenges

Abstract

“Big Data” is considered to be the fourth pillar of science nowadays and, as High Performance Computing (HPC) for modelling and numerical simulation, is crucial for science and industrial competitivity. The volume and the complexity of data do not stop growing and, in several areas, the volume and the complexity of data challenge our capacities to explore and to analyze them. This also explains why Artificial Intelligence attracts again nowadays a strong interest for data analysis with the very successful use of Deep Learning for example for pattern recognition but also for knowledge representation, decision-making automatic language translation, etc ... It is not anymore possible to dissociate HPC and “Big Data” i.e. the exploitation of the data arising from digital simulations (climate, combustion, fusion, astrophysics), large instruments (LHC, ITER, LSST, LOFAR, genomic platforms), ground or space systems of observation (seismology and geodesy RESIF, Euclid, WFIRST, GAIA, imaging and interferometry) or simply multiple devices of data acquisition (broadband sequencer, sensors’ networks, social networks, etc.). Processing such large amounts of data requires both significant processing capabilities and suitable methods for data analysis at large scale. Artificial intelligence techniques such as Deep Learning also require the use of supercomputers. In multiple scientific and socioeconomic domains, the volumetry and the variety of the existing data as well as the time constraints of calculation revealed new challenges. In a wide range of areas: in fundamental sciences (physics of high energy, fusion, earth and universe sciences, bio-computing, neurosciences, etc.), digital economy (business intelligence, Web, e-commerce, social networks, e-government, health, telecommunications and media), ground and air transport, financial markets, environment (climate, natural risks, energy resources, smart cities, connected house), security, industry (smart industry, customized products, design and production chain), new tools, scientific methods and new technologies are necessary. We will illustrate some of these issues using examples coming from the Informatics Research Institute of Toulouse (IRIT) and from CNRS.