Hadoop源码分析:Writable接口和WritableComparable接口

1、 Writable接口

由于Hadoop的MapReduce和HDFS都有通信的需求,需要对通信的对象进行序列化。Hadoop并没有采用Java的序列化,而是引入了它自己的系统。 org.apache.hadoop.io中定义了大量的可序列化对象,他们都实现了Writable接口。

1.1 Writable接口源码

package org.apache.hadoop.io;

import java.io.DataOutput;
import java.io.DataInput;
import java.io.IOException;
import org.apache.hadoop.classification.InterfaceAudience;
import org.apache.hadoop.classification.InterfaceStability;


@InterfaceAudience.Public
@InterfaceStability.Stable
public interface Writable {
  /** 
   * Serialize the fields of this object to <code>out</code>.
   * 
   */
  void write(DataOutput out) throws IOException;

  /** 
   * Deserialize the fields of this object from <code>in</code>.  
   * 
   */
  void readFields(DataInput in) throws IOException;
}

1.2 自定义Writable实现类

实现了Writable接口的一个典型例子如下,该代码实际上是在Writable接口源码的注释中。

public class MyWritable implements Writable {
       // Some data     
       private int counter;
        private long timestamp;

        public void write(DataOutput out) throws IOException {
         out.writeInt(counter);
         out.writeLong(timestamp);
       }

       public void readFields(DataInput in) throws IOException {
        counter = in.readInt();
         timestamp = in.readLong();
       }

      public static MyWritable read(DataInput in) throws IOException {
        MyWritable w = new MyWritable();
        w.readFields(in);
        return w;
       }
    }

2 WritableComparable接口

WritableComparable是Writable接口和java.lang.Comparable的一个子接口。

2.1 WritableComparable接口源码

package org.apache.hadoop.io;
import org.apache.hadoop.classification.InterfaceAudience;
import org.apache.hadoop.classification.InterfaceStability;

@InterfaceAudience.Public
@InterfaceStability.Stable
public interface WritableComparable<T> extends Writable, Comparable<T> {
}

2.2 WritableComparable一个典型实现类

 public class MyWritableComparable implements WritableComparable<MyWritableComparable> {
       // Some data
        private int counter;
        private long timestamp;

        public void write(DataOutput out) throws IOException {
          out.writeInt(counter);
          out.writeLong(timestamp);
        }

        public void readFields(DataInput in) throws IOException {
          counter = in.readInt();
          timestamp = in.readLong();
        }

       public int compareTo(MyWritableComparable o) {
          int thisValue = this.value;
          int thatValue = o.value;
          return (thisValue &lt; thatValue ? -1 : (thisValue==thatValue ? 0 : 1));
        }

      public int hashCode() {
         final int prime = 31;
         int result = 1;
         result = prime * result + counter;
         result = prime * result + (int) (timestamp ^ (timestamp &gt;&gt;&gt; 32));
         return result
       }
     }

2.3 IntWritable

IntWritable实现了WritableComparable。

package org.apache.hadoop.io;
import java.io.DataInput;
import java.io.DataOutput;
import java.io.IOException;
import org.apache.hadoop.classification.InterfaceAudience;
import org.apache.hadoop.classification.InterfaceStability;


@InterfaceAudience.Public
@InterfaceStability.Stable
public class IntWritable implements WritableComparable<IntWritable> {
  private int value;

  public IntWritable() {}

  public IntWritable(int value) { set(value); }

  /** Set the value of this IntWritable. */
  public void set(int value) { this.value = value; }

  /** Return the value of this IntWritable. */
  public int get() { return value; }

  @Override
  public void readFields(DataInput in) throws IOException {
    value = in.readInt();
  }

  @Override
  public void write(DataOutput out) throws IOException {
    out.writeInt(value);
  }

  /** Returns true iff <code>o</code> is a IntWritable with the same value. */
  @Override
  public boolean equals(Object o) {
    if (!(o instanceof IntWritable))
      return false;
    IntWritable other = (IntWritable)o;
    return this.value == other.value;
  }

  @Override
  public int hashCode() {
    return value;
  }

  /** Compares two IntWritables. */
  @Override
  public int compareTo(IntWritable o) {
    int thisValue = this.value;
    int thatValue = o.value;
    return (thisValue<thatValue ? -1 : (thisValue==thatValue ? 0 : 1));
  }

  @Override
  public String toString() {
    return Integer.toString(value);
  }

  /** A Comparator optimized for IntWritable. */ 
  public static class Comparator extends WritableComparator {
    public Comparator() {
      super(IntWritable.class);
    }

    @Override
    public int compare(byte[] b1, int s1, int l1,
                       byte[] b2, int s2, int l2) {
      int thisValue = readInt(b1, s1);
      int thatValue = readInt(b2, s2);
      return (thisValue<thatValue ? -1 : (thisValue==thatValue ? 0 : 1));
    }
  }

  static {                                        // register this comparator
    WritableComparator.define(IntWritable.class, new Comparator());
  }
}

本文参与腾讯云自媒体分享计划,欢迎正在阅读的你也加入,一起分享。

发表于

我来说两句

0 条评论
登录 后参与评论

相关文章

来自专栏鸿的学习笔记

抛弃丑陋,拥抱优雅--Pythonic的Pony ORM

Pony ORM是一个设计的相当精巧的ORM框架,可以让你用Pythonic的方式去处理表数据,并且把ER图的思想融合进代码里。现在就看Pony ORM吧!

2843
来自专栏GIS讲堂

判断是否是数字类

972
来自专栏向治洪

Android ormLite复杂条件查询

OrmLite要继承一个OrmLiteSqliteOpenHelper,通过OrmLiteSqliteOpenHelper实例的getDao方法可以获取一个Da...

2488
来自专栏乐沙弥的世界

PL/SQL --> 语言基础

PL/SQL是过程化的SQL语言,是ORACLE对SQL语言的扩展,在普通SQL语句的基础上增加了编程语言的特点。使得该语言不仅具有过程编程语

963
来自专栏积累沉淀

朋友开网店 做个抓取数据的小程序

朋友开网店需要填充初期的数据.  专门做了一个抓取数据的小程序.分享一下. private void button1_Click(object sender,...

2137
来自专栏JMCui

SQL优化一(SQL使用技巧)

1、行列转换:   decode(条件,值1,返回值1,值2,返回值2,...值n,返回值n,缺省值);   select decode(sign(变量1-变量...

3783
来自专栏Jackson0714

详解SQL集合运算

3498
来自专栏Hongten

java开发_mysql中获取数据库表描述_源码下载

我们需要获取表:test_table表的描述信息,然后把描述信息插入到表:data_element_config中记录结果

1462
来自专栏杨建荣的学习笔记

通过java程序抽取日志中的sql语句(r4笔记第4天)

今天在翻看以前的笔记时,发现自己在很早之前写过一个java程序,能够解析日志中的sql语句。 当时使用的环境是weblogic,日志目录下总是有几十上百个日志文...

2916
来自专栏康怀帅的专栏

MySQL 函数

函数相关内容。 字符函数 字符连接 CONCAT('a','b') 使用指定分隔符进行字符连接 CONCAT_WS('-','a','b') 数字格式化 FOR...

2824

扫码关注云+社区