Description

The map and reduce functions in MapReduce are easy to test in isolation, which is a consequence of their functional style. For known inputs, they produce known outputs. The following example is a program calculating the max temperature in Hadoop framework Unit test driver for mapper: Mockito is an open source testing framework for Java Then checks the output is the year and temperature reading It passes a weather record as input to the mapper  mock() To create a mock Context, we call Mockito’s mock() method (a static import), passing the class of the type we want to mock.   Map() Then we invoke the mapper’s map() method, which executes the code being tested. Verify() Here we verify that Context’s write() method was called with a Text object representing the year (1950) and an IntWritable representing the temperature (−1.1°C). Unit test driver for reducer: Source code for reducer Two parameter for the reducer to compare with A tool interface that supports handling of generic command-line options. A driver test the job By adding a driver class which can run the job in custom configuration, the job can be tested locally ToolRunner can be used to run classes implementing Tool interface Details of the Hadoop installation number of maps and reduces number of job submissions number of tasktracker nodes currently available cluster’s capacity number of map and reduce slots available across the cluster number of available slots per node number of tasktrackers that have been blacklisted by the jobtracker Sections for running, (successfully) completed, and failed jobs. Each of these sections has a table of jobs, with a row per job that shows the job’s ID, Links to the jobtracker’s logs, and the jobtracker’s history: information on all the jobs that the jobtracker has run. job owner and name how long the job has been running for “Num Tasks” shows the total number of map and reduce tasks for this job (a row for each) The job file is t

AI summary