Fall 2019 Big Data Processing and Analytics Vassilis Christophides christop@csd.uoc.gr http://www.csd.uoc.gr/~hy562 University of Crete, Fall 2019 1 Fall 2019 The Data Avalanche: From Science to Business 2 1 1 Fall 2019 Shifting Paradigm in Sciences https://aip.scitation.org/doi/10.1063/1.4946894 3 Fall 2019 Data-driven Discovery  Data-driven discovery is revolutionizing scientific exploration as well as engineering innovations From hypothesis driven to hypothesis generating R. Leland, R. Murphy, B. Hendrickson, K. Yelick, J. Johnson, J. Berry Large-Scale Data Analytics & its Relationship to Simulation Jan. 20144 2 2 From “Data Door” to “Data Rich” Scientific Research Astronomy: LSST Physics: LHC Fall 2019 Oceanography Biology: Sequencing Sociology: The Web Precision Medicine Neuroscience: EEG, fMRI Sports  Data deluge spans biology, climate, cosmology, materials, physics, … M. Franklin Big Data Software: What’s Next? (and what do we have to say about it?) VLDB 2017 5 Fall 2019 New Research Methods  Simulation Data: Increasing level of  Experimental Data: Light sources, simulation detail and duration, as well genome sequencing, next generation as, model size by orders of magnitude! ARM radars, sky surveys, neurosensing and stimulation, …  New research methods depend on coupling computation and experiment as well as on integrating data across sources and/or types 6 3 3 Fall 2019 Large Synoptic Survey Telescope (LSST) 100-200 Petabyte image archive 20-40 Petabyte database catalog  LSST will take more than 800 panoramic images each night recording the entire visible sky twice each week  Ten-year time series (~2020-2030) imaging of the night sky – mapping the Universe ! 3.2 billion-pixel camera 8.4-meter diameter primary mirror = 10 square degrees! www.lsst.org 7 Fall 2019 First Image of a Black Hole  Captured by the Event Horizon telescope (EHT), an NSF funded network of eight radio telescopes spanning locations f