Graph Processing with Neo4j Lecture BigData Analytics Julian M. Kunkel julian.kunkel@googlemail.com University of Hamburg / German Climate Computing Center (DKRZ) 2016-12-09 Disclaimer: Big Data software is constantly updated, code samples may be outdated.
Overview Cypher Query Language (CQL) Interfaces Architecture Summary Outline 1 Overview 2 Cypher Query Language (CQL) 3 Interfaces 4 Architecture 5 Summary Julian M. Kunkel Lecture BigData Analytics, 2016 2 / 37
Overview Cypher Query Language (CQL) Interfaces Architecture Summary Neo4j [31, 32] Graph database written in Java Supports ACID transaction semantics One server scales to billions of nodes/relationships Performance: Millions of node traversals/s High availability (and performance) through clustering Declarative query language Cypher (instead of SQL) Note: Very loose connection to Hadoop ecosystem Prepare data in, e.g., HBASE for batch import in Neo4j Suboptimal import of Millions of nodes can take days Schema-optional: You can use a schema To gain performance To improve modeling, e.g., via constraints Many interfaces to the graph database Julian M. Kunkel Lecture BigData Analytics, 2016 3 / 37
Overview Cypher Query Language (CQL) Interfaces Architecture Summary Graph Data Model (Slight Changes for Neo4j) Nodes: Entity Edges: Relationship between two nodes They have a direction Property: (key, value) Attributes describe relationships/nodes Key is string, the value has a type Label: Organize nodes into groups Definitions for queries Path: One or more nodes with connecting relationships Traversal: Navigates through a graph to find paths Source: What’s a Graph Database [31] Julian M. Kunkel Lecture BigData Analytics, 2016 4 / 37
Overview Cypher Query Language (CQL) Interfaces Architecture Summary Example Graph Use-Cases Movie and actors data [31] Movies: label, title, released date, tagline People: label, name, born (date, optional) Relationships ACTED_IN from actor to movie, roles (list of played chars) DIRECTED from director to movie Source: Online Course: Introduction to Graph Databases and Neo4j [31] Julian M. Kunkel Lecture BigData Analytics, 2016 5 / 37
Overview Cypher Query Language (CQL) Interfaces Architecture Summary Converting RDBMS to Graphs A1 B1 A2 B2 A3 B3 B4 Consider three tables A,B,C B5 Relations between rows C1 B6 (foreign keys) become edges C2 B7 C3 Source: RDBMS. The Neo4j Manual v2.2.5 [33] C3 A1 A2 C1 A3 C2 B2 B6 B1 B4 B5 B3 B7 Source: Graph Database as RDBMS. The Neo4j Manual v2.2.5 [33] Julian M. Kunkel Lecture BigData Analytics, 2016 6 / 37
Overview Cypher Query Language (CQL) Interfaces Architecture Summary Converting Key-Value Store Models to Graphs V1 K2 K1 K2 V2 K3 K1 K3 V3 K1 Source: Key-Value Store. The Neo4j Manual v2.2.5 [33] K2 V2 K1 V1 V3 K3 Source: Graph Database as Key-Value Store. The Neo4j Manual [33] Julian M. Kunkel Lecture BigData Analytics, 2016 7 / 37
Overview Cypher Query Language (CQL) Interfaces Architecture Summary Converting the Document Store Model to Graphs D1 D2 S1 S3 S2 D2/S2 V1 D1/S1 V4 V3 V2 Source: Document Store. The Neo4j Manual v2.2.5 [33] D=Document, S=Subdocument, V=Value, X/Y=reference to a subdocument in another document D1 V1 V2 D2 S3 S1 S2 V3 V4 Source: Graph Database as Document Store. The Neo4j Manual v2.2.5 [33] Julian M. Kunkel Lecture BigData Analytics, 2016 8 / 37
Overview Cypher Query Language (CQL) Interfaces Architecture Summary Neo4j Case Success Studies [31] For the logistics company Accenture Use case: Dynamic parcel routing (5 million parcels/day) With Neo4j: Routing of packets online, i.e., where to load a parcel For the communication company SFR Use case: Prioritize hardware replacement to minimize downtime Run automated “what if” analysis to ensure resilience With Neo4j: Loading data from > 30 systems works; easier analysis model Source: [36] Julian M. Kunkel Lecture BigData Analytics, 2016 9 / 37
Overview Cypher Query Language (CQL) Interfaces Architecture Summary 1 Overview 2 Cypher Query Language (CQL) 3 Interfaces 4 Architecture 5 Summary Julian M. Kunkel Lecture BigData Analytics, 2016 10 / 37
Overview Cypher Query Language (CQL) Interfaces Architecture Summary Cypher Query Language Basics [31] Declarative query language for formulating graph queries Allows query and/or update of the graph Each part of a query must be read-only or write-only A query consists of multiple clauses Transactions can span multiple queries Supports: variables, expressions1, operators, comments Supports collections (list, dictionary) Provides functions for aggregation, collections, strings, math 1Handling missing values with NULL is possible, see http://neo4j.com/docs/stable/cypher-working-with-null.html Julian M. Kunkel Lecture BigData Analytics, 2016 11 / 37
Overview Cypher Query Language (CQL) Interfaces Architecture Summary Cypher Query Language [33] Syntax: specifying graph structures via patterns Node Anonymous node: () Named node: (x), the variable x is used to refer to it Node with a specific label (class): (x : label) Relationship Named relationship: −[r]− > Typed relationship: −[r : t]− > Two nodes with a relationship: (a) − [r]− > (b) Properties can be specified in {}, i.e., (x {name:"Hans"}) A pattern combines several nodes/relations Julian M. Kunkel Lecture BigData Analytics, 2016 12 / 37
Overview Cypher Query Language (CQL) Interfaces Architecture Summary Cypher Query Language Read Clauses [33] LOAD CSV: read data from a CSV file, can be used for importing MATCH: search for something (returns a relational table) DISTINCT keyword: Avoid replicates (e.g., returning a node twice) OPTIONAL MATCH: optional relationship like SQL outer join WHERE: Filtering Supports regex matching of strings Pattern predicates restrict the graph’s shape Aggregation functions Automatic grouping on all non-aggregated columns sum, avg, percentileDisc, count e.g., count(*), count(DISTINCT X) collect(x): creates a list of all values Julian M. Kunkel Lecture BigData Analytics, 2016 13 / 37
Overview Cypher Query Language (CQL) Interfaces Architecture Summary Cypher Query Language Write Clauses [33] CREATE: an element or relation MERGE: Create or lookup (CREATE + MATCH) SET: Modify/Add data/labels REMOVE: remove labels and properties DELETE: remove graph elements Julian M. Kunkel Lecture BigData Analytics, 2016 14 / 37
Overview Cypher Query Language (CQL) Interfaces Architecture Summary Cypher Query Language: Interactive Session 1 # Create a star graph 2 $ CREATE (c) FOREACH (x IN range(1,6)| CREATE (l),(c)-[:X]->(l)) RETURN id(c); 3 id 4 0 5 Updated the graph - created 7 nodes and 6 relationships 6 7 # Count the number of nodes 8 $ MATCH (n) RETURN count(n); # since we have not defined any restriction, all nodes 9 count(n) 10 7 11 12 # Count relationships based on their type 13 $ MATCH ()-[r]->() RETURN type(r), count(*); 14 type(r) count(*) 15 X 6 16 17 # Set the center node’s name property to CENTER 18 $ MATCH (n) WHERE id(n) = 184 SET n.name = "CENTER"; 19 20 # Clean the database 21 $ MATCH (n) OPTIONAL MATCH (n)-[r]-() DELETE n, r; Julian M. Kunkel Lecture BigData Analytics, 2016 15 / 37
Overview Cypher Query Language (CQL) Interfaces Architecture Summary Cypher Query Language General Clauses [33] FOREACH(< col >|< op >): iterates through a collection, apply op RETURN: return the subgraph/table Usually you can convert those into a response table AS x: rename column to x ORDER BY x (ASC|DESC): sorting SKIP, LIMIT X: paginate UNION: compose statements WITH: a barrier for a pipeline of multiple statements Example: retrieve the top entries by a criteria and join it with other data Allows also to combine read-only and write-only parts Aggregated results must pass through a WITH clause UNWIND: expand a collection into a sequence of rows USING: instruction to use/avoid indexes Julian M. Kunkel Lecture BigData Analytics, 2016 16 / 37
Overview Cypher Query Language (CQL) Interfaces Architecture Summary Cypher Query Language [33]: Selection of Functions id(): the node id timestamp(): a timestamp label(): the node label upper(), lower(): change case range(l,u): return a collection with numbers from l to u length(x): size of a collection keys(x): keys of a dictionary coalesce(x, y): use property x if available, else y nodes(path), rels(path), length(path) Julian M. Kunkel Lecture BigData Analytics, 2016 17 / 37
Overview Cypher Query Language (CQL) Interfaces Architecture Summary Cypher Query Language: Examples [33] 1 # Return a collection 2 $ RETURN [1, 2, 3] 3 4 # Return a string with a row name of X 5 $ RETURN "BigData" as X 6 7 # Return a dictionary 8 $ RETURN {key1 : 2, key2 : "test"} 9 10 # Return a list of x^3 where x is an even number 11 $ RETURN [x IN range(1,10) WHERE x % 2 = 0 | x^3] AS result 12 13 # populate a table 14 $ CREATE (matrix1:Movie { title : ’The Matrix’, year : ’1999-03-31’ }) 15 $ CREATE (keanu:Actor { name:’Keanu Reeves’ }) 16 $ CREATE (keanu)-[:ACTS_IN { role : ’Neo’ }]->(matrix1) 17 18 # Create actor keanu if he does not exist 19 $ MERGE (keanu:Actor { name:’Keanu Reeves’ }) 20 21 # Eliminate duplicates from a collection 22 $ WITH [1,1,2,2] AS coll UNWIND coll AS x WITH DISTINCT x RETURN collect(x) AS SET 23 # [1,2] Julian M. Kunkel Lecture BigData Analytics, 2016 18 / 37
Overview Cypher Query Language (CQL) Interfaces Architecture Summary Cypher Query Language: Examples [33] 1 # Read a table from a (large) CSV 2 USING PERIODIC COMMIT 3 LOAD CSV WITH HEADERS FROM ’http://neo4j.com/docs/2.2.5/csv/artists-with-headers.csv’ AS ↪→ line 4 CREATE (:Artist { name: line.Name, year: toInt(line.Year)}) 5 6 MATCH (a:Movie { title: ’Wall Street’ }) 7 OPTIONAL MATCH (a)-->(x) 8 RETURN x 9 10 # return a movie and all properties 11 MATCH (movie:Movie { title: ’The Matrix’ }) 12 RETURN movie; 13 14 # return certain attributes 15 MATCH (movie:Movie { title: ’The Matrix’ }) 16 RETURN movie.title, movie.year; 17 18 # show all actors sorted by name 19 MATCH (actor:Actor) 20 RETURN actor ORDER BY actor.name; 21 22 # all actors whose name end with s 23 MATCH (actor:Actor) 24 WHERE actor.name =~ ".*s$" 25 RETURN actor.name; Julian M. Kunkel Lecture BigData Analytics, 2016 19 / 37
Overview Cypher Query Language (CQL) Interfaces Architecture Summary Cypher Query Language: Examples [33] 1 # List all nodes together with their relationsships 2 MATCH (n)-[r]->(m) RETURN n AS from , r AS ‘->‘, m AS to; 3 4 # Return number of movies for actors acting in "The Matrix" 5 MATCH (:Movie { title: "The Matrix" })<-[:ACTS_IN]-(actor)-[:ACTS_IN]->(movie) 6 RETURN movie.title, collect(actor.name), count(*) AS count 7 ORDER BY count DESC ; 8 9 # Filtering 10 MATCH (p:Person)-[r:ACTED_IN]->(m:Movie) 11 WHERE p.name =~ "K.+" OR m.released > 2000 OR "Neo" IN r.roles 12 RETURN p,r,m 13 14 # Filtering based on graph structure 15 # Here: Search for people that are actors in any movie but never directed any movie 16 MATCH (p:Person)-[:ACTED_IN]->(m) 17 WHERE NOT (p)-[:DIRECTED]->() 18 RETURN p,m 19 20 # Identify how often actors and directors worked together 21 MATCH (actor:Person)-[:ACTED_IN]->(movie:Movie)<-[:DIRECTED]-(director:Person) 22 RETURN actor,director,count(*) AS collaborations Julian M. Kunkel Lecture BigData Analytics, 2016 20 / 37
Hiển thị 20 trang đầu. Tải tài liệu để xem đầy đủ.
Chưa có bình luận nào. Hãy là người đầu tiên!
Bình luận (0)
Chưa có bình luận nào. Hãy là người đầu tiên!