Debug Hadoop (Kiểm thử và gỡ lỗi các chương trình MapReduce trong Hadoop)

UN
Uni24h
0 lượt tải
14
0

Nội dung tài liệu

Trang 2

The map and reduce functions in MapReduce are easy to test in isolation, which is a consequence of their functional style. For known inputs, they produce known outputs. The following example is a program calculating the max temperature in Hadoop framework

Trang 3

Unit test driver for mapper: Mockito is an open source testing framework for Java Then checks the output is the year and temperature reading It passes a weather record as input to the mapper

Trang 4

 mock() To create a mock Context, we call Mockito’s mock() method (a static import), passing the class of the type we want to mock.  Map() Then we invoke the mapper’s map() method, which executes the code being tested.  Verify() Here we verify that Context’s write() method was called with a Text object representing the year (1950) and an IntWritable representing the temperature (−1.1°C).

Trang 6

A tool interface that supports handling of generic command-line options. A driver test the job By adding a driver class which can run the job in custom configuration, the job can be tested locally ToolRunner can be used to run classes implementing Tool interface

Trang 7

Details of the Hadoop installation • number of maps and reduces • number of job submissions • number of tasktracker nodes currently available cluster’s capacity • number of map and reduce slots available across the cluster • number of available slots per node • number of tasktrackers that have been blacklisted by the jobtracker Sections for running, (successfully) completed, and failed jobs. Each of these sections has a table of jobs, with a row per job that shows the job’s ID, Links to the jobtracker’s logs, and the jobtracker’s history: information on all the jobs that the jobtracker has run.

Trang 8

job owner and name The job file is the consolidated configuration file for the job, containing how long the job has been all the properties and their values that running for were in effect during the job run “Num Tasks” shows the total number of map and reduce tasks for this job (a row for each)

Trang 9

The reduce completion graph is divided into the three phases of the reduce task: 1. copy (when the map outputs are being transferred to the reduce’s tasktracker), 2. sort (when the reduce inputs are being merged), 3. reduce (when the reduce function is being run to produce the final output).

Hiển thị 9 trang đầu. Tải tài liệu để xem đầy đủ.

Bình luận (0)

Đăng nhập để tham gia bình luận

Chưa có bình luận nào. Hãy là người đầu tiên!