Hadoop源码分析:Writable接口和WritableComparable接口

1、 Writable接口

由于Hadoop的MapReduce和HDFS都有通信的需求,需要对通信的对象进行序列化。Hadoop并没有采用Java的序列化,而是引入了它自己的系统。 org.apache.hadoop.io中定义了大量的可序列化对象,他们都实现了Writable接口。

1.1 Writable接口源码

package org.apache.hadoop.io;

import java.io.DataOutput;
import java.io.DataInput;
import java.io.IOException;
import org.apache.hadoop.classification.InterfaceAudience;
import org.apache.hadoop.classification.InterfaceStability;


@InterfaceAudience.Public
@InterfaceStability.Stable
public interface Writable {
  /** 
   * Serialize the fields of this object to <code>out</code>.
   * 
   */
  void write(DataOutput out) throws IOException;

  /** 
   * Deserialize the fields of this object from <code>in</code>.  
   * 
   */
  void readFields(DataInput in) throws IOException;
}

1.2 自定义Writable实现类

实现了Writable接口的一个典型例子如下,该代码实际上是在Writable接口源码的注释中。

public class MyWritable implements Writable {
       // Some data     
       private int counter;
        private long timestamp;

        public void write(DataOutput out) throws IOException {
         out.writeInt(counter);
         out.writeLong(timestamp);
       }

       public void readFields(DataInput in) throws IOException {
        counter = in.readInt();
         timestamp = in.readLong();
       }

      public static MyWritable read(DataInput in) throws IOException {
        MyWritable w = new MyWritable();
        w.readFields(in);
        return w;
       }
    }

2 WritableComparable接口

WritableComparable是Writable接口和java.lang.Comparable的一个子接口。

2.1 WritableComparable接口源码

package org.apache.hadoop.io;
import org.apache.hadoop.classification.InterfaceAudience;
import org.apache.hadoop.classification.InterfaceStability;

@InterfaceAudience.Public
@InterfaceStability.Stable
public interface WritableComparable<T> extends Writable, Comparable<T> {
}

2.2 WritableComparable一个典型实现类

 public class MyWritableComparable implements WritableComparable<MyWritableComparable> {
       // Some data
        private int counter;
        private long timestamp;

        public void write(DataOutput out) throws IOException {
          out.writeInt(counter);
          out.writeLong(timestamp);
        }

        public void readFields(DataInput in) throws IOException {
          counter = in.readInt();
          timestamp = in.readLong();
        }

       public int compareTo(MyWritableComparable o) {
          int thisValue = this.value;
          int thatValue = o.value;
          return (thisValue &lt; thatValue ? -1 : (thisValue==thatValue ? 0 : 1));
        }

      public int hashCode() {
         final int prime = 31;
         int result = 1;
         result = prime * result + counter;
         result = prime * result + (int) (timestamp ^ (timestamp &gt;&gt;&gt; 32));
         return result
       }
     }

2.3 IntWritable

IntWritable实现了WritableComparable。

package org.apache.hadoop.io;
import java.io.DataInput;
import java.io.DataOutput;
import java.io.IOException;
import org.apache.hadoop.classification.InterfaceAudience;
import org.apache.hadoop.classification.InterfaceStability;


@InterfaceAudience.Public
@InterfaceStability.Stable
public class IntWritable implements WritableComparable<IntWritable> {
  private int value;

  public IntWritable() {}

  public IntWritable(int value) { set(value); }

  /** Set the value of this IntWritable. */
  public void set(int value) { this.value = value; }

  /** Return the value of this IntWritable. */
  public int get() { return value; }

  @Override
  public void readFields(DataInput in) throws IOException {
    value = in.readInt();
  }

  @Override
  public void write(DataOutput out) throws IOException {
    out.writeInt(value);
  }

  /** Returns true iff <code>o</code> is a IntWritable with the same value. */
  @Override
  public boolean equals(Object o) {
    if (!(o instanceof IntWritable))
      return false;
    IntWritable other = (IntWritable)o;
    return this.value == other.value;
  }

  @Override
  public int hashCode() {
    return value;
  }

  /** Compares two IntWritables. */
  @Override
  public int compareTo(IntWritable o) {
    int thisValue = this.value;
    int thatValue = o.value;
    return (thisValue<thatValue ? -1 : (thisValue==thatValue ? 0 : 1));
  }

  @Override
  public String toString() {
    return Integer.toString(value);
  }

  /** A Comparator optimized for IntWritable. */ 
  public static class Comparator extends WritableComparator {
    public Comparator() {
      super(IntWritable.class);
    }

    @Override
    public int compare(byte[] b1, int s1, int l1,
                       byte[] b2, int s2, int l2) {
      int thisValue = readInt(b1, s1);
      int thatValue = readInt(b2, s2);
      return (thisValue<thatValue ? -1 : (thisValue==thatValue ? 0 : 1));
    }
  }

  static {                                        // register this comparator
    WritableComparator.define(IntWritable.class, new Comparator());
  }
}

本文参与腾讯云自媒体分享计划,欢迎正在阅读的你也加入,一起分享。

发表于

我来说两句

0 条评论
登录 后参与评论

相关文章

来自专栏冷冷

【微信开发】getAccessToken 和 getJsapi_ticket缓存支持

AccessToken   JsapiTicket 获取这个两个参数的接口每天都有上线,所以我们应该使用缓存把它们保存起来,减少对这两个接口的调用 示例:  /...

19410
来自专栏数据之美

Hadoop Mapper 阶段将数据直接从 HDFS 导入 Hbase

数据源格式如下: 20130512 1 -1 -1 13802 1 2013-05-12 07:26:22 20130512 1 -1 -1 13802 ...

1746
来自专栏Sorrower的专栏

Android用5种方式实现自定义计时器, 哪种才是你的菜?

802
来自专栏浪淘沙

MapReduce操作实例

要点:有一个combiner方法,可以在执行完map时调用,从而对数据进行先一步的处理,降低Reduce的IO压力。

723
来自专栏LanceToBigData

Hadoop(十五)MapReduce程序实例

一、统计好友对数(去重) 1.1、数据准备 joe, jon joe , kia joe, bob joe ,ali kia, ...

2368
来自专栏积累沉淀

Mapreduce和HBase新版本整合之WordCount计数案例

先计数单词数量存到hdfs文件上,这个是以前的就做过的 package com.my.myhnase.mapreduce; import java.io.IO...

18210
来自专栏计算机视觉与深度学习基础

基于hadoop的社交网络三角形计数

图的三角形计数问题是一个基本的图计算问题,是很多复杂网络分析(比如社交网络分析) 的基础。目前图的三角形计数问题已经成为了 Spark 系统中 GraphX 图...

2225
来自专栏菩提树下的杨过

Hadoop: MapReduce2的几个基本示例

1) WordCount  这个就不多说了,满大街都是,网上有几篇对WordCount的详细分析 http://www.sxt.cn/u/235/blog/58...

2326
来自专栏码匠的流水账

聊聊rocketmq的PullConsumerImpl

io/openmessaging/rocketmq/consumer/PullConsumerImpl.java

301
来自专栏码匠的流水账

reactor-netty中TcpClient的create过程

本文主要研究一下reactor-netty中TcpClient的create的过程

541

扫码关注云+社区