Big Data Analytics Big Data Analytics B. Distributed Storage / B.3 NoSQL Databases Lars Schmidt-Thieme Information Systems and Machine Learning Lab (ISMLL) Institute for Computer Science University of Hildesheim, Germany Lars Schmidt-Thieme, Information Systems and Machine Learning Lab (ISMLL), University of Hildesheim, Germany 1 / 32
Big Data Analytics Syllabus Tue. 9.4. (1) 0. Introduction A. Parallel Computing Tue. 16.4. (2) A.1 Threads Tue. 23.4. (3) A.2 Message Passing Interface (MPI) Tue. 30.4. (4) A.3 Graphical Processing Units (GPUs) B. Distributed Storage Tue. 7.5. (5) B.1 Distributed File Systems Tue. 14.5. (6) B.2 Partioning of Relational Databases Tue. 21.5. (7) B.3 NoSQL Databases C. Distributed Computing Environments Tue. 28.5. (8) C.1 Map-Reduce Tue. 4.6. — — Pentecoste Break — Tue. 11.6. (9) C.2 Resilient Distributed Datasets (Spark) Tue. 18.6. (10) C.3 Computational Graphs (TensorFlow) D. Distributed Machine Learning Algorithms Tue. 25.6. (11) D.1 Distributed Stochastic Gradient Descent Tue. 2.7. (12) D.2 Distributed Matrix Factorization Tue. 9.7. (13) Questions and Answers Lars Schmidt-Thieme, Information Systems and Machine Learning Lab (ISMLL), University of Hildesheim, Germany 1 / 32
Big Data Analytics Outline 1. Introduction 2. Key-Value Stores 3. Document Databases 4. Document Databases: Partitioning 5. Other NoSQL Databases Lars Schmidt-Thieme, Information Systems and Machine Learning Lab (ISMLL), University of Hildesheim, Germany 1 / 32
Big Data Analytics 1. Introduction Outline 1. Introduction 2. Key-Value Stores 3. Document Databases 4. Document Databases: Partitioning 5. Other NoSQL Databases Lars Schmidt-Thieme, Information Systems and Machine Learning Lab (ISMLL), University of Hildesheim, Germany 1 / 32
Big Data Analytics 1. Introduction Relational vs Big-Data Technologies I Structure: I relational data bases: data is structured. I big data applications: data often is raw. I Process: I relational data bases: initially created for transactional processing. I big data applications: analytical processing. I Entities: I relational data bases: big if it has many entities (rows) of a kind. I big data applications: the number of entities is not necessary high, the amount of information that exist for entities may be large. I horizontal scaling: I relational data bases: costs for adding new nodes is high. I big data applications: scaling should be accomplished inexpensively by adding new nodes. Lars Schmidt-Thieme, Information Systems and Machine Learning Lab (ISMLL), University of Hildesheim, Germany 1 / 32
Big Data Analytics 1. Introduction NoSQL Databases I A NoSQL or Not Only SQL database provides a mechanism for storage and retrieval of data that is modeled in means other than the tabular relations used in relational databases. I Motivations for this approach include I simplicity of design and I horizontal scaling. I The data structure differs from the RDBMS, and therefore I some operations are faster in NoSQL, and I some operations are faster in RDBMS. I Most NoSQL stores lack true ACID transactions. I ACID: atomicity, consistency, isolation, durability Lars Schmidt-Thieme, Information Systems and Machine Learning Lab (ISMLL), University of Hildesheim, Germany 2 / 32
Big Data Analytics 1. Introduction NoSQL Databases / Types and Implementations I Key-value: RedisOS, Amazon DynamoDB, MemcachedOS I Document: MongoDBOS, Amazon DynamoDB, CouchbaseOS I Graph: Neo4JOS, MS CosmosDB, OrientDBOS I Column: CassandraOS, HBaseOS, MS CosmosDB I Object: Caché, Versant Object Database, ObjectStore, Db4oOS Note: Popularity ranking of DBMS by type: https://db-engines.com/en/ranking/; OS=open source Lars Schmidt-Thieme, Information Systems and Machine Learning Lab (ISMLL), University of Hildesheim, Germany 3 / 32
Big Data Analytics 1. Introduction BASE vs ACID I CAP-theorem: in the presence of network partitions, a DBMBs has to choose between I Consistency: all reads always return the last write — then clients cannot read/write during a network partition. I Availability: clients always can read/write — then what they receive might not be consistent at all times I BASE design: I Basically available: the system aims for high availability. I Soft state: the state of the system may change over time, even without input. (due to the eventual consistency model). I Eventual consistency: the system may not be consistent at all times, but will become consistent over time. Lars Schmidt-Thieme, Information Systems and Machine Learning Lab (ISMLL), University of Hildesheim, Germany 4 / 32
Big Data Analytics 1. Introduction http://blog.scottlogic.com Lars Schmidt-Thieme, Information Systems and Machine Learning Lab (ISMLL), University of Hildesheim, Germany 5 / 32
Big Data Analytics 2. Key-Value Stores Outline 1. Introduction 2. Key-Value Stores 3. Document Databases 4. Document Databases: Partitioning 5. Other NoSQL Databases Lars Schmidt-Thieme, Information Systems and Machine Learning Lab (ISMLL), University of Hildesheim, Germany 6 / 32
Big Data Analytics 2. Key-Value Stores Key–Value stores I Key–Value stores use the associative array (dictionary) as their fundamental data model. I Data is represented as a collection of key–value pairs. I One of the simplest non-trivial data models. Example: { "Great Expectations": "John", "Pride and Prejudice": "Alice", "Wuthering Heights": "Alice" } Lars Schmidt-Thieme, Information Systems and Machine Learning Lab (ISMLL), University of Hildesheim, Germany 6 / 32
Big Data Analytics 3. Document Databases Outline 1. Introduction 2. Key-Value Stores 3. Document Databases 4. Document Databases: Partitioning 5. Other NoSQL Databases Lars Schmidt-Thieme, Information Systems and Machine Learning Lab (ISMLL), University of Hildesheim, Germany 7 / 32
Big Data Analytics 3. Document Databases Document Databases I Data abstraction: I relational databases: "relations" (= "tables"). I document databases: "document". I Documents are (possibly nested) dictionaries. I like Python dicts. I Documents are not required to have all the same fields (aka sections, slots, parts). I Documents are schemaless. I Documents are identified by an ID, e.g., I marked by a special type ObjectiId, I stored by a special key _id. I References are modeled by foreign keys. I or avoided by using embedded documents. Lars Schmidt-Thieme, Information Systems and Machine Learning Lab (ISMLL), University of Hildesheim, Germany 7 / 32
Big Data Analytics 3. Document Databases Example 1 / Schema-less 1 { 1 { 2 _id: ObjectId(7df78ad8902c), 2 _id: ObjectId(5df78ad8902c), 3 FirstName: "Bob", 3 FirstName: "Jonathan", 4 Age: 35, 4 Age: 37, 5 Address: "5 Oak St.", 5 Address: "15 Wanamassa Point Road", 6 Hobby: " sailing " 6 Languages: [ ’ English ’, ’German’ ] 7 } 7 } Lars Schmidt-Thieme, Information Systems and Machine Learning Lab (ISMLL), University of Hildesheim, Germany 8 / 32
Big Data Analytics 3. Document Databases Example 2 / Foreign Keys 1 { 2 _id: ObjectId(5df78ad8902c), 3 FirstName: "Jonathan", 4 Age: 37, 5 Address: "15 Wanamassa Point Road", 6 Children : [ ObjectId(5df78ad89020), ObjectId(5df78ad89021), 7 ObjectId(5df78ad89022), ObjectId(5df78ad89023) ] 8 },{ 9 _id: ObjectId(5df78ad89020), 10 FirstName: "Michael", 11 Age: 10, 12 },{ 13 _id: ObjectId(5df78ad89021), 14 FirstName: " Jennifer ", 15 Age: 8, 16 },{ 17 _id: ObjectId(5df78ad89022), 18 FirstName: "Samantha", 19 Age: 5, 20 },{ 21 _id: ObjectId(5df78ad89023), 22 FirstName: "Elena", 23 Age: 2, 24 } Lars Schmidt-Thieme, Information Systems and Machine Learning Lab (ISMLL), University of Hildesheim, Germany 9 / 32
Big Data Analytics 3. Document Databases Example 3 / Embedded Documents 1 { 2 _id: ObjectId(5df78ad8902c), 3 FirstName: "Jonathan", 4 Age: 37, 5 Address: "15 Wanamassa Point Road", 6 Children : { 7 { FirstName: "Michael", Age: 10 }, 8 { FirstName: " Jennifer ", Age: 8 }, 9 { FirstName: "Samantha", Age: 5 }, 10 { FirstName: "Elena", Age: 2} 11 } 12 } Lars Schmidt-Thieme, Information Systems and Machine Learning Lab (ISMLL), University of Hildesheim, Germany 10 / 32
Big Data Analytics 3. Document Databases Organization I Documents are addressed in the database via a unique key. I Documents can be retrieved by their I key I content I Documents are organized through I Collections I Tags I Non-visible Metadata I Directory hierarchies I Buckets Lars Schmidt-Thieme, Information Systems and Machine Learning Lab (ISMLL), University of Hildesheim, Germany 11 / 32
Big Data Analytics 3. Document Databases RDBMS vs. Document Database Terminology relational database document database Database Database Table Collection Tuple/Row Document Column Field Primary key Primary key (e.g., default key _id in mongodb) Foreign key Foreign key Lars Schmidt-Thieme, Information Systems and Machine Learning Lab (ISMLL), University of Hildesheim, Germany 12 / 32
Big Data Analytics 3. Document Databases Inserting Documents I insert allows to insert a new document. I insertMany allows to insert many documents. 1 db. inventory .insertMany([ 2 { item: " journal ", qty : 25, size : { h: 14, w: 21, uom: "cm" }, status: "A" }, 3 { item: "notebook", qty: 50, size : { h: 8.5, w: 11, uom: "in" }, status : "A" }, 4 { item: "paper", qty : 100, size : { h: 8.5, w: 11, uom: "in" }, status : "D" }, 5 { item: "planner", qty : 75, size : { h: 22.85, w: 30, uom: "cm" }, status: "D" }, 6 { item: "postcard", qty : 45, size : { h: 10, w: 15.25, uom: "cm" }, status: "A" } 7 ]); Note: Most examples taken from MongoDB [2017]. Lars Schmidt-Thieme, Information Systems and Machine Learning Lab (ISMLL), University of Hildesheim, Germany 13 / 32
Big Data Analytics 3. Document Databases Query Documents (1/3) I all documents of a collection: 1 db. inventory . find () 1 { "_id" : ObjectId("59116d5340229e45bb5eea9a"), "item" : "journal", "qty" : 25, " size " : { "h" : 14, "w" 2 { "_id" : ObjectId("59116d5340229e45bb5eea9b"), "item" : "notebook", "qty" : 50, " size " : { "h" : 8.5, "w 3 { "_id" : ObjectId("59116d5340229e45bb5eea9c"), "item" : "paper", "qty" : 100, " size " : { "h" : 8.5, "w" 4 { "_id" : ObjectId("59116d5340229e45bb5eea9d"), "item" : "planner", "qty" : 75, " size " : { "h" : 22.85, " 5 { "_id" : ObjectId("59116d5340229e45bb5eea9e"), "item" : "postcard", "qty" : 45, " size " : { "h" : 10, "w I all documents with a property: 1 db. inventory . find ({ status : "D"}) 1 { "_id" : ObjectId("59116d5340229e45bb5eea9c"), "item" : "paper", "qty" : 100, " size " : { "h" : 8.5, "w" 2 { "_id" : ObjectId("59116d5340229e45bb5eea9d"), "item" : "planner", "qty" : 75, " size " : { "h" : 22.85, " I all documents with several properties: 1 db. inventory . find ({ status : "D", qty: 100}) 1 { "_id" : ObjectId("59116d5340229e45bb5eea9c"), "item" : "paper", "qty" : 100, " size " : { "h" : 8.5, "w" Lars Schmidt-Thieme, Information Systems and Machine Learning Lab (ISMLL), University of Hildesheim, Germany 14 / 32
显示前 20 页。下载资料查看完整内容。
暂无评论。快来抢沙发吧！
评论 (0)
暂无评论。快来抢沙发吧！