java – 错误的密钥类:文本不是IntWritable

前端之家收集整理的这篇文章主要介绍了java – 错误的密钥类:文本不是IntWritable前端之家小编觉得挺不错的,现在分享给大家,也给大家做个参考。
这可能看起来像一个愚蠢的问题,但我没有在我的mapreduce代码中看到我的类型中的问题为hadoop

正如问题中所述,问题是它期望IntWritable,但我在reducer的collector.collect中传递了一个Text对象.

我的作业配置有以下映射器输出类:

  1. conf.setMapOutputKeyClass(IntWritable.class);
  2. conf.setMapOutputValueClass(IntWritable.class);

以下减速机输出类:

  1. conf.setOutputKeyClass(Text.class);
  2. conf.setOutputValueClass(IntWritable.class);

我的映射类具有以下定义:

  1. public static class Reduce extends MapReduceBase implements Reducer<IntWritable,IntWritable,Text,IntWritable>

具有所需功能

  1. public void reduce(IntWritable key,Iterator<IntWritable> values,OutputCollector<Text,IntWritable> output,Reporter reporter)

然后当我打电话时失败:

  1. output.collect(new Text(),new IntWritable());

我是相当新的map reduce,但所有类型似乎都匹配,它编译但是然后在该行上失败,说它期望IntWritable作为reduce类的键.如果重要的话我使用的是0.21版本的Hadoop

这是我的地图类:

  1. public static class Map extends MapReduceBase implements Mapper<LongWritable,IntWritable> {
  2. private IntWritable node = new IntWritable();
  3. private IntWritable edge = new IntWritable();
  4.  
  5. public void map(LongWritable key,Text value,OutputCollector<IntWritable,Reporter reporter) throws IOException {
  6. String line = value.toString();
  7. StringTokenizer tokenizer = new StringTokenizer(line);
  8.  
  9. while (tokenizer.hasMoreTokens()) {
  10. node.set(Integer.parseInt(tokenizer.nextToken()));
  11. edge.set(Integer.parseInt(tokenizer.nextToken()));
  12. if(node.get() < edge.get())
  13. output.collect(node,edge);
  14. }
  15. }
  16. }

和我的减少类:

  1. public static class Reduce extends MapReduceBase implements Reducer<IntWritable,IntWritable> {
  2.  
  3. IntWritable $= new IntWritable(Integer.MAX_VALUE);
  4. Text keyText = new Text();
  5.  
  6. public void reduce(IntWritable key,Reporter reporter) throws IOException {
  7. ArrayList<IntWritable> valueList = new ArrayList<IntWritable>();
  8.  
  9. //outputs original edge pair as key and $for value
  10. while (values.hasNext()) {
  11. IntWritable value = values.next();
  12. valueList.add(value);
  13. keyText.set(key.get() + "," + value.get());
  14. output.collect(keyText,$);
  15. }
  16.  
  17. //outputs all the 2 length pairs
  18. for(int i = 0; i < valueList.size(); i++)
  19. for(int j = i+1; i < valueList.size(); j++)
  20. output.collect(new Text(valueList.get(i).get() + "," + valueList.get(j).get()),key);
  21. }
  22. }

和我的工作配置:

  1. JobConf conf = new JobConf(Triangles.class);
  2. conf.setJobName("mapred1");
  3.  
  4. conf.setMapOutputKeyClass(IntWritable.class);
  5. conf.setMapOutputValueClass(IntWritable.class);
  6.  
  7. conf.setOutputKeyClass(Text.class);
  8. conf.setOutputValueClass(IntWritable.class);
  9.  
  10. conf.setMapperClass(Map.class);
  11. conf.setCombinerClass(Reduce.class);
  12. conf.setReducerClass(Reduce.class);
  13.  
  14. conf.setInputFormat(TextInputFormat.class);
  15. conf.setOutputFormat(TextOutputFormat.class);
  16.  
  17. FileInputFormat.setInputPaths(conf,new Path(args[0]));
  18. FileOutputFormat.setOutputPath(conf,new Path("mapred1"));
  19.  
  20. JobClient.runJob(conf);

解决方法

您的问题是您将Reduce类设置为组合器
  1. conf.setCombinerClass(Reduce.class);

组合器在映射阶段运行,它们需要发出相同的键/值类型(在您的情况下为IntWriteable,IntWritable)删除这一行,你应该没问题

猜你在找的Java相关文章