Debug Hadoop (Kiểm thử và gỡ lỗi các chương trình MapReduce trong Hadoop)
- ページ数
- 14
- 形式
- PPTX
- サイズ
- 758 KB
- Trường
- University of Maryland
- 閲覧数
- 0
- コメント
- 0
- Lượt tải
- 0
プレビューを生成中...
Tài liệu hướng dẫn kiểm thử và gỡ lỗi các chương trình MapReduce trong Hadoop, bao gồm kiểm thử mapper, reducer, công cụ ToolRunner và giám sát job.
- ドキュメント名
- Debug Hadoop (Kiểm thử và gỡ lỗi các chương trình MapReduce trong Hadoop)
- 学校 / コース
- University of Maryland · Khai phá dữ liệu
- 内容
- Tài liệu hướng dẫn cách gỡ lỗi và kiểm thử các thành phần MapReduce trong Hadoop, bao gồm kiểm thử đơn vị cho mapper và reducer, sử dụng Mockito, và cách theo dõi, phân tích log của job.
- 目次
- このドキュメントに明確な目次はありません。
- ページ数
- 14 ページ
- アップロード者
- Uni24h
説明
Trích nội dung tài liệu
The map and reduce functions in MapReduce are easy to test in isolation, which is a consequence of their functional style. For known inputs, they produce known outputs. The following example is a program calculating the max temperature in Hadoop framework Unit test driver for mapper: Mockito is an open source testing framework for Java Then checks the output is the year and temperature reading It passes a weather record as input to the mapper mock() To create a mock Context, we call Mockito’s mock() method (a static import), passing the class of the type we want to mock. Map() Then we invoke the mapper’s map() method, which executes the code being tested. Verify() Here we verify that Context’s write() method was called with a Text object representing the year (1950) and an IntWritable representing the temperature (−1.1°C). Unit test driver for reducer: Source code for reducer Two parameter for the reducer to compare with A tool interface that supports handling of generic command-line options. A driver test the job By adding a driver class which can run the job in custom configuration, the job can be tested locally ToolRunner can be used to run classes implementing Tool interface Details of the Hadoop installation number of maps and reduces number of job submissions number of tasktracker nodes currently available cluster’s capacity number of map and reduce slots available across the cluster number of available slots per node number of tasktrackers that have been blacklisted by the jobtracker Sections for running, (successfully) completed, and failed jobs. Each of these sections has a table of jobs, with a row per job that shows the job’s ID, Links to the jobtracker’s logs, and the jobtracker’s history: information on all the jobs that the jobtracker has run. job owner and name how long the job has been running for “Num Tasks” shows the total number of map and reduce tasks for this job (a row for each) The job file is t
よくある質問
このドキュメントは無料ですか?
はい。「Debug Hadoop (Kiểm thử và gỡ lỗi các chương trình MapReduce trong Hadoop)」は無料です。ログインして「ダウンロード」をクリックするだけで、元のファイルを取得できます。
このドキュメントは何ページありますか?
このドキュメントは 14 ページあります(Khai phá dữ liệu コース用)。ダウンロードする前にオンラインでプレビューできます。
ダウンロードする前にプレビューできますか?
はい。このページにあるオンラインリーダーでドキュメントをプレビューし、その後ダウンロードするかどうかを決めることができます。
Debug Hadoop (Kiểm thử và gỡ lỗi các chương trình MapReduce trong Hadoop)
プレビューを生成中...
Trích nội dung tài liệu
The map and reduce functions in MapReduce are easy to test in isolation, which is a consequence of their functional style. For known inputs, they produce known outputs. The following example is a program calculating the max temperature in Hadoop framework Unit test driver for mapper: Mockito is an open source testing framework for Java Then checks the output is the year and temperature reading It passes a weather record as input to the mapper mock() To create a mock Context, we call Mockito’s mock() method (a static import), passing the class of the type we want to mock. Map() Then we invoke the mapper’s map() method, which executes the code being tested. Verify() Here we verify that Context’s write() method was called with a Text object representing the year (1950) and an IntWritable representing the temperature (−1.1°C). Unit test driver for reducer: Source code for reducer Two parameter for the reducer to compare with A tool interface that supports handling of generic command-line options. A driver test the job By adding a driver class which can run the job in custom configuration, the job can be tested locally ToolRunner can be used to run classes implementing Tool interface Details of the Hadoop installation number of maps and reduces number of job submissions number of tasktracker nodes currently available cluster’s capacity number of map and reduce slots available across the cluster number of available slots per node number of tasktrackers that have been blacklisted by the jobtracker Sections for running, (successfully) completed, and failed jobs. Each of these sections has a table of jobs, with a row per job that shows the job’s ID, Links to the jobtracker’s logs, and the jobtracker’s history: information on all the jobs that the jobtracker has run. job owner and name how long the job has been running for “Num Tasks” shows the total number of map and reduce tasks for this job (a row for each) The job file is t
- ドキュメント名
- Debug Hadoop (Kiểm thử và gỡ lỗi các chương trình MapReduce trong Hadoop)
- 学校 / コース
- University of Maryland · Khai phá dữ liệu
- 内容
- Tài liệu hướng dẫn cách gỡ lỗi và kiểm thử các thành phần MapReduce trong Hadoop, bao gồm kiểm thử đơn vị cho mapper và reducer, sử dụng Mockito, và cách theo dõi, phân tích log của job.
- 目次
- このドキュメントに明確な目次はありません。
- ページ数
- 14 ページ
- アップロード者
- Uni24h
コメント (0)
まだコメントはありません。最初のコメントを書きましょう!
Relational Data Processing on MapReduce (Xử lý dữ liệu quan hệ trên MapReduce)
IoT Data Analytics (Phân tích dữ liệu trong Internet vạn vật)
Colocation 2 (Khám phá mẫu colocation trong dữ liệu không gian) - Zhe Jiang
Big Data Processing and Analytics Intro (Xử lý và phân tích dữ liệu lớn)
Basic association analysis (Chap 6) (Thuật toán trong phân tích kết hợp)
Tổng hợp Đề Toán 5 - Luyện thi vào Lớp 6 - CLB EMath
Bài giảng vật lý đại cương (Chương 3) - Đỗ Ngọc Uấn
Chương 8.Nguyên tử - Vật lý đại cương 3 - TS.Nguyễn Thị Trang
Chương 7.Cơ học lượng tử - Vật lý đại cương 3 - TS.Nguyễn Thị Trang
Chương 6.Quang học lượng tử - Vật lý đại cương 3 - TS.Nguyễn Thị Trang

コメント (0)
まだコメントはありません。最初のコメントを書きましょう!