Hadoop Programming Overview MapReduce Types Input Formats Output Formats Serialization Job http://hadoop.apache.org/docs/r2.2.0/api/or g/apache/hadoop/mapreduce/packagesummary.html Mapper<KEYIN,VALUEIN,KEYOUT,VALUEOUT> Maps input key/value pairs to a set of intermediate key/value pairs. Maps are the individual tasks which transform input records into a intermediate records. The transformed intermediate records need not be of the same type as the input records. A given input pair may map to zero or many output pairs. The Hadoop Map-Reduce framework spawns one map task for each InputSplit generated by the InputFormat for the job. The framework first calls setup(org.apache.hadoop.mapreduce.Mapper.Context), followed by map(Object, Object, Context) for each key/value pair in the InputSplit. Finally cleanup(Context) is called. http://hadoop.apache.org/docs/r2.2.0/api/org/apache/hadoop/mapreduce/Mapper.ht ml public static class TokenizerMapper extends Mapper<Object, Text, Text, IntWritable>{ private final static IntWritable one = new IntWritable(1); private Text word = new Text(); public void map(Object key, Text value, Context context ) throws IOException, InterruptedException { StringTokenizer itr = new StringTokenizer(value.toString()); while (itr.hasMoreTokens()) { word.set(itr.nextToken()); context.write(word, one); } } } What is Writable? Hadoop defines its own “box” classes for strings (Text), integers (IntWritable), etc. All values are instances of Writable All keys are instances of WritableComparable Writable A serializable object which implements a simple, efficient, serialization protocol, based on DataInput and DataOutput. Any key or value type in the Hadoop Map-Reduce framework implements this interface. Implementations typically implement a static read(DataInput) method which constructs a new instance, calls readFields(DataInput) and returns the instance. http://hadoop.apache.org/docs/r2.2.0/api/or g/apache/hadoop

Lire le document entier