Document Object Storage with MongoDB Lecture BigData Analytics Julian M. Kunkel julian.kunkel@googlemail.com University of Hamburg / German Climate Computing Center (DKRZ) 2016-12-16 Disclaimer: Big Data software is constantly updated, code samples may be outdated. Document Object Storage Architecture Interfaces Outline 1 Document Object Storage 2 Architecture 3 Interfaces Julian M. Kunkel Lecture BigData Analytics, 2016 2 / 16 Document Object Storage Architecture Interfaces Data Model Documents contain semi-structured data (JSON, XML) Each document can contain data with other structures Addressing to lookup documents are implementation specific e.g., bucket/document key, (sub) collections, hierarchical namespace References between documents are possible Example technology: MongoDB, Couchbase, DocumentDB D1 D2 S1 D2/S2 S3 V1 D1/S1 S2 V4 V3 V2 Source: Document Store. The Neo4j Manual v2.2.5 [33] D=Document, S=Subdocument, V=Value, X/Y=reference to a subdocument in another document Julian M. Kunkel Lecture BigData Analytics, 2016 3 / 16 Document Object Storage Architecture Interfaces MongoDB [11] Open-source document database High-performant and horizontally scalable for clusters Interfaces: interactive shell: mongo, REST, C, Python, ... Connector for Hadoop for reading/writing to MongoDB Data model Database: as usual, defines permissions Document: BSON object (binary JSON) – consisting of subdocuments Primary key: _id field (manually set or automatically filled) 1 2 3 4 "_id" : ObjectId("43459bc2341bc14b1b41b124"), "students" : [ # subdocument { "name" : "Julian", "id" : 4711, "birth" : ISODate("2000-10-01")}, { "name" : "Hans", "id" : 4712, "birth", ... } ] Collection: like a table of documents Keys: name, _id field Documents can have individual schemas Support for indexes on fields (and compound fields) Document references via object ids Julian M. Kunkel Lecture BigData Analytics, 2016 4 / 16 Document Object Storage