This post originated from an RSS feed registered with Scala Buzz by David Bernard. Original Post: A Scalable Language, and a Scalable Framework Feed Title: Scala Blog Feed URL: http://www.scala-blogs.org/feeds/posts/default?alt=rss Feed Description: In an effort to realize the "grow together" spirit, the developers of lift have teamed up with other great minds in the Scala community to bring you Scala-Blogs.org. Our mission is to strengthen the community by sharing our experiences and knowledge so that others can learn from our mistakes and triumph along with our successes. At Scala-Blogs.org you will find tutorials and articles written by a growing host of enthusiasts, each with a different background and area of expertise. Latest Scala Buzz Posts Latest Scala Buzz Posts by David Bernard Latest Posts From Scala Blog

public class WordCount {
  public static class Map extends MapReduceBase 
       implements Mapper<LongWritable, IntWritable> {
    private final static IntWritable one = new IntWritable(1);
    private Text word = new Text();

    public void map(LongWritable key, Text value,
                    OutputCollector<Text, IntWritable> output,
                    Reporter reporter) throws IOException {
      String line = value.toString();
      StringTokenizer tokenizer = new StringTokenizer(line);
      while (tokenizer.hasMoreTokens()) {
        word.set(tokenizer.nextToken());
        output.collect(word, one);
      }
    }
  }

  public static class Reduce extends MapReduceBase 
      implements Reducer<Text, IntWritable> {

    public void reduce(Text key, Iterator<IntWritable> values,
                       OutputCollector<Text, IntWritable> output,
                       Reporter reporter) throws IOException {
      int sum = 0;
      while (values.hasNext()) {
        sum += values.next().get();
      }
      output.collect(key, new IntWritable(sum));
    }
  }

  public static void main(String[] args) throws Exception {
    JobConf conf = new JobConf(WordCount.class);
    conf.setJobName("wordcount");

    conf.setOutputKeyClass(Text.class);
    conf.setOutputValueClass(IntWritable.class);

    conf.setMapperClass(Map.class);
    conf.setCombinerClass(Reduce.class);
    conf.setReducerClass(Reduce.class);

    conf.setInputFormat(TextInputFormat.class);
    conf.setOutputFormat(TextOutputFormat.class);

    FileInputFormat.setInputPaths(conf, new Path(args[0]));
    FileOutputFormat.setOutputPath(conf, new Path(args[1]));

    JobClient.runJob(conf);
 }
}

object WordCount {
 def main(args : Array[String]) {
   val (h,remainingArgs) = Hadoop.fromArgs(args,new Path("output"));
   val words = for( (offset,line) <- h.loadLines(remainingArgs);
                    word <- line.split(" \n\r\t\f"))
                    yield(word,1);

   val sums = words.hreduce{ (word,iter) =>
     (word,iter.reduceLeft(_+_));
   }
   sums.elements foreach println
 }
}

1. Syntactic Support for Map: By treating for loops as syntactic sugar for map and its cousins, Scala allows you to create new classes that interact with the primitive looping construct in intuitive ways.
   
2. Strong Type System: Scala's type system not only takes a lot of the boiler plate away from the code, but it makes library writer's lives easier by making much richer type information available at compile time. In SMR, the type system is used to automatically select the correct "input format type" and figure out what the types of the Maps and the Reduces are.
3. Higher-Order and Anonymous Functions: The combination of functions that take other functions and the ability to create new functions on the fly easily and succinctly obviates a lot of the boiler plate associated with defining a new "Mapper" and "Reducer" for every task. Instead, the compiler automatically creates the functions that perform the task for you.