Page 1
Intro to Apache Spark ! http://databricks.com/ download slides: http://cdn.liber118.com/workshop/itas_workshop.pdf
Contenu du document
Page 3
Intro: Online Course Materials Best to download the slides to your laptop: cdn.liber118.com/workshop/itas_workshop.pdf Be sure to complete the course survey: http://goo.gl/QpBSnR In addition to these slides, all of the code samples are available on GitHub gists: • gist.github.com/ceteri/f2c3486062c9610eac1d • gist.github.com/ceteri/8ae5b9509a08c08a1132 • gist.github.com/ceteri/11381941
Page 4
Intro: Success Criteria By end of day, participants will be comfortable with the following: • open a Spark Shell • use of some ML algorithms • explore data sets loaded from HDFS, etc. • review Spark SQL, Spark Streaming, Shark • review advanced topics and BDAS projects • follow-up courses and certification • developer community resources, events, etc. • return to workplace and demo use of Spark!
Page 5
Intro: Preliminaries • intros – what is your background? • who needs to use AWS instead of laptops? • PEM key, if needed? See tutorial: Connect to Your Amazon EC2 Instance from Windows Using PuTTY
Page 7
Installation: Let’s get started using Apache Spark, in just four easy steps… spark.apache.org/docs/latest/ (for class, please copy from the USB sticks)
Page 8
Step 1: Install Java JDK 6/7 on MacOSX or Windows oracle.com/technetwork/java/javase/downloads/ jdk7-downloads-1880260.html • follow the license agreement instructions • then click the download for your OS • need JDK instead of JRE (for Maven, etc.) (for class, please copy from the USB sticks)
Page 9
Step 1: Install Java JDK 6/7 on Linux this is much simpler on Linux…! sudo apt-get -y install openjdk-7-jdk
Page 10
Step 2: Download Spark we’ll be using Spark 1.0.0 see spark.apache.org/downloads.html 1. download this URL with a browser 2. double click the archive file to open it 3. connect into the newly created directory (for class, please copy from the USB sticks)
Page 11
Step 3: Run Spark Shell we’ll run Spark’s interactive shell… ./bin/spark-shell! then from the “scala>” REPL prompt, let’s create some data… val data = 1 to 10000
Page 12
Step 4: Create an RDD create an RDD based on that data… val distData = sc.parallelize(data)! then use a filter to select values less than 10… distData.filter(_ < 10).collect()
Page 13
Step 4: Create an RDD create an val distData = sc.parallelize(data) then use a filter to select values less than 10… d Checkpoint: what do you get for results? gist.github.com/ceteri/ f2c3486062c9610eac1d#file-01-repl-txt
Page 14
Installation: Optional Downloads: Python For Python 2.7, check out Anaconda by Continuum Analytics for a full-featured platform: store.continuum.io/cshop/anaconda/
Page 15
Installation: Optional Downloads: Maven Java builds later also require Maven, which you can download at: maven.apache.org/download.cgi
Page 18
Spark Deconstructed: Log Mining Example // load error messages from a log into memory! // then interactively search for various patterns! // https://gist.github.com/ceteri/8ae5b9509a08c08a1132! ! // base RDD! val lines = sc.textFile("hdfs://...")! ! // transformed RDDs! val errors = lines.filter(_.startsWith("ERROR"))! val messages = errors.map(_.split("\t")).map(r => r(1))! messages.cache()! ! // action 1! messages.filter(_.contains("mysql")).count()! ! // action 2! messages.filter(_.contains("php")).count()
Page 19
Spark Deconstructed: Log Mining Example We start with Spark running on a cluster… submitting code to be evaluated on it: Worker Worker Driver Worker
Affichage des 19 premières pages. Téléchargez le document pour le voir en entier.
Commentaires (0)
Aucun commentaire pour le moment. Soyez le premier !