Data Models & Processing Lecture BigData Analytics Julian M. Kunkel julian.kunkel@googlemail.com University of Hamburg / German Climate Computing Center (DKRZ) 2016-10-28 Disclaimer: Big Data software is constantly updated, code samples may be outdated. Data: Terminology Data Models & Processing Big Data Data Models Technology Summary Outline 1 Data: Terminology 2 Data Models & Processing 3 Big Data Data Models 4 Technology 5 Summary Julian M. Kunkel Lecture BigData Analytics, 2016 2 / 33 Data: Terminology Data Models & Processing Big Data Data Models Technology Summary Basic Considerations About Storing Big Data Analysis requires efficient (real-time) processing of data New data is constantly coming (Velocity of Big Data) How do we technically ingest the data? In respect to performance and data quality How can we update our derived data (and conclusions)? Incremental updates vs. (partly) re-computation algorithms Storage and data management techniques are needed How do we map the logical data to physical hardware and organize it? How can we diagnose causes for problems with data (e.g., inaccuracies)? Management of data Idea: Store facts (truth) and never change them (data lake idea) Data value may degrade over time, garbage clean old data Raw data is usually considered to be immutable Implies that an update of (raw) data is not necessary Create ad-hoc models for representing the data Julian M. Kunkel Lecture BigData Analytics, 2016 3 / 33 Data: Terminology Data Models & Processing Big Data Data Models Technology Summary Terminology Data [1, 10] Raw data: collected information that is not derived from other data Derived data: data produced with some computation/functions View: presents derived data to answer specific questions Convenient for users (only see what you need) + faster than re-computation Convenient for administration (e.g., manage permissions) Data access can be optimized Dealing with unstructured data We ne