Page 1
Apache Spark Performance Troubleshooting at Scale: Challenges, Tools and Methods Luca Canali, CERN #EUdev2
Page 2
About Luca • Computing engineer and team lead at CERN IT – Hadoop and Spark service, database services – Joined CERN in 2005 • 17+ years of experience with database services – Performance, architecture, tools, internals – Sharing information: blog, notes, code • @LucaCanaliDB – http://cern.ch/canali #EUdev2 2
Page 4
Apache Spark @ • Spark is a popular component for data processing – Deployed on four production Hadoop/YARN clusters • Aggregated capacity (2017): ~1500 physical cores, 11 PB – Adoption is growing. Key projects involving Spark: • Analytics for accelerator controls and logging • Monitoring use cases, this includes use of Spark streaming • Analytics on aggregated logs • Explorations on the use of Spark for high energy physics Link: http://cern.ch/canali/docs/BigData_Solutions_at_CERN_KT_Forum_20170929.pdf #EUdev2 4
Page 5
Motivations for This Work • Understanding Spark workloads – Understanding technology (where are the bottlenecks, how much do Spark jobs scale, etc?) – Capacity planning: benchmark platforms • Provide our users with a range of monitoring tools • Measurements and troubleshooting Spark SQL – Structured data in Parquet for data analytics – Spark-ROOT (project on using Spark for physics data) #EUdev2 5
Page 6
Outlook of This Talk • Topic is vast, I will just share some ideas and lessons learned • How to approach performance troubleshooting, benchmarking and relevant methods • Data sources and tools to measure Spark workloads, challenges at scale • Examples and lessons learned with some key tools #EUdev2 6
Page 7
Challenges • Just measuring performance metrics is easy • Producing actionable insights requires effort and preparation – Methods on how to approach troubleshooting performance – How to gather relevant data • Need to use the right tools, possibly many tools • Be aware of the limitations of your tools – Know your product internals: there are many “moving parts” – Model and understand root causes from effects #EUdev2 7
Page 8
Anti-Pattern: The Marketing Benchmark 12 • The over-simplified System B is 5x better benchmark graph 10 than System A !? • Does not tell you why B 8 is better than A 6 • To understand, you need 4 more context and root 2 cause analysis SOME METRIC (HIGHER IS BETTER) 0 System A System B #EUdev2 8
Page 10
Adapt Answer to Circumstances • Which one is faster? • 20x 10x 1x • Actually, it depends.. #EUdev2 10
Page 11
Active Benchmarking • Example: use TPC-DS benchmark as workload generator – Understand and measure Spark SQL, optimizations, systems performance, etc 3000 T P C D S W O R K L O AD - D AT A S E T S I Z E : 1 0 T B - Q U E R Y S E T V 1 . 4 4 2 0 C O R E S , E X E C U T O R M E M O R Y P E R C O R E 5 G 2500 MIN_Exec MAX_Exec AVG_Exec_Time_sec 2000 1500 seconds 1000 500 Query Execution Time (Latency) in 0 qSs… Query #EUdev2 11
Page 12
Troubleshooting by Understanding • Measure the workload – Use all relevant tools – Not a “black box”: instrument code where is needed • Be aware of the blind spots – Missing tools, measurements hard to get, etc • Make a mental model – Explain the observed performance and bottlenecks – Prove it or disprove it with experiment • Summary: – Be data driven, no dogma, produce insights #EUdev2 12
Page 13
Actionable Measurement Data • You want to find answers to questions like – What is my workload doing? – Where is it spending time? – What are the bottlenecks (CPU, I/O)? – Why do I measure the {latency/throughput} that I measure? • Why not 10x better? #EUdev2 13
Page 14
Measuring Spark • Distributed system, parallel architecture – Many components, complexity increases when running at scale – Optimizing a component does not necessarily optimize the whole #EUdev2 14
Page 15
Spark and Monitoring Tools • Spark instrumentation – Web UI – REST API – Eventlog – Executor/Task Metrics – Dropwizard metrics library • Complement with – OS tools – For large clusters, deploy tools that ease working at cluster-level • https://spark.apache.org/docs/latest/monitoring.html #EUdev2 15
Page 16
Web UI • Info on Jobs, Stages, Executors, Metrics, SQL,.. – Start with: point web browser driver_host, port 4040 #EUdev2 16
Page 18
Web UI Event Timeline • Event Timeline – show task execution details by activity and time #EUdev2 18
Page 19
REST API – Spark Metrics • History server URL + /api/v1/applications • http://historyserver:18080/api/v1/applicati ons/application_1507881680808_0002/s tages #EUdev2 19
Page 20
EventLog – Stores Web UI History • Config: – spark.eventLog.enabled=true – spark.eventLog.dir = <path> • JSON files store info displayed by Spark History server – You can read the JSON files with Spark task metrics and history with custom applications. For example sparklint. – You can read and analyze event log files using the Dataframe API with the Spark SQL JSON reader. More details at: https://github.com/LucaCanali/Miscellaneous/tree/master/Spark_Notes #EUdev2 20
Affichage des 20 premières pages. Téléchargez le document pour le voir en entier.
Commentaires (0)
Aucun commentaire pour le moment. Soyez le premier !