Apache Spark Performance Troubleshooting at Scale: Challenges, Tools and Methods Luca Canali, CERN #EUdev2 About Luca Computing engineer and team lead at CERN IT Hadoop and Spark service, database services Joined CERN in 2005 17+ years of experience with database services Performance, architecture, tools, internals Sharing information: blog, notes, code @LucaCanaliDB – http://cern.ch/canali #EUdev2 2 CERN and the Large Hadron Collider Largest and most powerful particle accelerator #EUdev2 3 Apache Spark @ Spark is a popular component for data processing Deployed on four production Hadoop/YARN clusters Aggregated capacity (2017): ~1500 physical cores, 11 PB Adoption is growing. Key projects involving Spark: Analytics for accelerator controls and logging Monitoring use cases, this includes use of Spark streaming Analytics on aggregated logs Explorations on the use of Spark for high energy physics Link: http://cern.ch/canali/docs/BigData_Solutions_at_CERN_KT_Forum_20170929.pdf #EUdev2 4 Motivations for This Work Understanding Spark workloads Understanding technology (where are the bottlenecks, how much do Spark jobs scale, etc?) Capacity planning: benchmark platforms Provide our users with a range of monitoring tools Measurements and troubleshooting Spark SQL Structured data in Parquet for data analytics Spark-ROOT (project on using Spark for physics data) #EUdev2 5 Outlook of This Talk Topic is vast, I will just share some ideas and lessons learned How to approach performance troubleshooting, benchmarking and relevant methods Data sources and tools to measure Spark workloads, challenges at scale Examples and lessons learned with some key tools #EUdev2 6 Challenges Just measuring performance metrics is easy Producing actionable insights requires effort and preparation Methods on how to approach troubleshooting performance How to gather relevant data Need to use the right tools, poss