Trang 2
Agenda • About speaker and itversity • Categorization of enterprise applications • Typical data integration architecture • Challenges with conventional technologies • Big Data eco system • Data Storage - Distributed file system and databases • Data Processing - Distributed computing frameworks • Data Ingestion - Open source tools • Data Visualization - BI tools as well as custom • Role of Apache and other companies • Different Certifications in Big Data • Resources for deep dive into Big Data • Job roles in Big Data
Trang 3
About me • IT Professional with 13+ years of experience in vast array of technologies including Oracle, Java/J2EE, Data Warehousing, ETL, BI as well as Big Data • Website: http://www.itversity.com • YouTube: https://www.youtube.com/c/itversityin • LinkedIn Profile: https://in.linkedin.com/in/durga0gadiraju • Facebook: https://www.facebook.com/itversity • Twitter: https://twitter.com/itversity • Google Plus: https://plus.google.com/+itversityin • Github: https://github.com/dgadiraju • Meetup: – https://www.meetup.com/Hyderabad-Technology-Meetup/ (local hyderabad meetup) – https://www.meetup.com/itversityin/ (typically online)
Trang 4
Categorization of Enterprise applications Customer Analytics Enterprise applications Decision Operational Support Traditionally enterprise applications can be broadly categorized into Operational and Decision support systems. Lately new set of applications such as Customer Analytics is gaining momentum (eg: YouTube Channel for different categories of users)
Trang 5
Enterprise Applications • Most of the traditional applications are considered monolith (n-tier architecture) – Monoliths are typically built on RDBMS databases such as Oracle • Modern applications use micro services and considered to be polyglot * Now we have choice of different types of databases (we will see later)
Trang 6
Enterprise applications Take the example use case (eCommerce platform) • Operational – Transactional – check out – Not transactional – recommendation engine • Decision support – sales trends • Customer analytics – categories in which customers have spent money * Data from transactional systems need to be integrated to non transactional, decision support, customer analytics etc
Trang 7
Data Integration • Data integration can be categorized into – Real time – Batch • Traditionally when there are no customer analytics and recommendation engines, we used to have – ODS (compliance and single source of truth) – EDW (Facts and dimensions to support reports) – ETL (to perform transformations and load data into EDW) – BI (to visualize and publish reports)
Trang 8
Data Integration (Current Architecture) EDW/ODS Visualization/ OLTP ODS Reporting Closed Reporting Main Frames Data Integration (ETL/Real Time) Decision Support XML Target system External apps (eg; EDW) Source(s)
Trang 9
Data Integration - Technologies • Batch ETL – Informatica, Data Stage etc • Real time data integration – Goldengate, Shareplex etc • ODS – Oracle, MySQL etc • EDW/MPP – Teradata, Greenplum etc • BI Tools – Cognos, Business Objects etc
Trang 10
Current Scenario - Challenges • Almost all operational systems are using relational databases (RDBMS like Oracle). – RDBMS are originally designed for Operational and transactional. • Not linearly scalable. – Transactions – Data integrity • Expensive • Predefined Schema • Data processing do not happen where data is stored (storage layer) – no data locality – Some processing happens at database server level (SQL) – Some processing happens at application server level (Java/.net) – Some processing happens at client/browser level (Java Script) • Almost all Data Warehouse appliances are expensive and not very flexible for customer analytics and recommendation engines
Trang 11
Evolution of Databases Now we have many choices of Databases • Relational Databases (Oracle, Informix, Sybase, MySQL etc) • Datawarehouse and MPP appliances (Teradata, Greenplum etc) • NoSQL Databases (Cassandra, HBase, MongoDB etc) • In memory Databases (Gemfire, Coherence etc) • Search based Databases (Elastic Search, Solr etc) • Batch processing frameworks (Map Reduce, Spark etc) • Graph Databases (Neo4j) * Modern applications need to be polyglot (different modules need different category of databases)
Trang 12
Big Data eco system – History • Started with Google search engine • Google’s use case is different fromenterprises – Crawl web pages – Index based on key words – Return search results • As conventional database technologies does not scale, them implemented – GFS (Distributed file system) – Google Map Reduce (Distributed processing engine) – Google Big Table (Distributed indexed table)
Trang 13
Big Data eco system - myths • Big Data is Hadoop • Big Data eco system can only solve problems with very large data sets • Big Data is cheap • Big Data provide variety of tools and can solve problems quickly • Big Data is a technology • Big Data is Data Science – Data Scientist need to have specialized mathematical skills – Domain knowledge – Minimal technology orientation – Data Science it self is separate domain - if required Big Data technologies can be used * Often people have unrealistic expectations on Big Data technologies
Trang 14
Big Data eco system – Characteristics • Distributed storage – Fault tolerance (RAID is replaced by replication) • Distributed computing/processing – Data locality (code goes to data) • Scalability (almost linear) • Low cost hardware (commodity) • Low licensing costs * Low cost hardware and software does not mean that Big Data is cheap for enterprises
Trang 15
Big Data eco system Big Data eco system of tools can be categorized into • Distributed file systems – HDFS – Cloud storage (s3/Azure blob) • Distributed processing engines – Map Reduce – Spark • Distributed databases (operational) – NoSQL databases (HBase, Cassandra) – Search databases (Elastic Search)
Trang 16
Big Data eco system - Evolution After successfully building search engine in new technologies, Google have published white papers • Distributed file system – GFS • Distributed processing Engine – Map Reduce • Distributed database – Big Table * Development of Big Data technologies such as Hadoop is started with these white papers
Trang 17
Data Storage • Data storage options in Big Data eco systems • Distributed file systems (streaming and batch access) – HDFS – Cloud storage • Distributed Databases (random access - distributed indexed tables) – Cassandra – HBase – MongoDB – Solr
Trang 18
Data Ingestion Data ingestion strategies are defined by sources from which data is pulled and sinks where data is stored • Sources – Relational Databases – Non relational Databases – Streaming web logs – Flat files • Sinks – HDFS – Relational or Non relational Databases – Data processing frameworks
Trang 19
Data Ingestion • Sqoop is used to get data from relational databases • Flume and/or Kafka is used to read data from web logs • Spark streaming, Storm, Flink etc are used to process data from Flume and/or Kafka before loading data into sinks
Trang 20
Data processing - Batch • I/O based – Map Reduce – Hive, Pig are wrappers on top of map reduce • In memory – Spark – Spark Data Frames is wrapper on top of core spark • As part of data processing typically we focus on transformations such as – Aggregations – Joins – Sorting – Ranking
Hiển thị 20 trang đầu. Tải tài liệu để xem đầy đủ.
Bình luận (0)
Chưa có bình luận nào. Hãy là người đầu tiên!