聊聊jump consistent hash

本文主要简介一下jump Consistent hash。

jump consistent hash

jump consistent hash是一致性哈希的一种实现,论文见A Fast, Minimal Memory, Consistent Hash Algorithm 经典的一致性哈希算法来自Consistent Hashing and Random Trees: Distributed Caching Protocols for Relieving Hot Spots on the World Wide Web jump consistent hash与之的主要区别是节点可以扩容,但是不会移除节点。

算法代码

int32_t JumpConsistentHash(uint64_t key, int32_t num_buckets) {
    int64_t b = -1, j = 0;
    while (j < num_buckets) {
        b = j;
        key = key * 2862933555777941757ULL + 1;
        j = (b + 1) * (double(1LL << 31) / double((key >> 33) + 1));
    }
    return b;
}

java实现

guava里头有个现成的实现 guava-22.0-sources.jar!/com/google/common/hash/Hashing.java

/**
   * Assigns to {@code hashCode} a "bucket" in the range {@code [0, buckets)}, in a uniform manner
   * that minimizes the need for remapping as {@code buckets} grows. That is, {@code
   * consistentHash(h, n)} equals:
   *
   * <ul>
   * <li>{@code n - 1}, with approximate probability {@code 1/n}
   * <li>{@code consistentHash(h, n - 1)}, otherwise (probability {@code 1 - 1/n})
   * </ul>
   *
   * <p>This method is suitable for the common use case of dividing work among buckets that meet the
   * following conditions:
   *
   * <ul>
   * <li>You want to assign the same fraction of inputs to each bucket.
   * <li>When you reduce the number of buckets, you can accept that the most recently added buckets
   * will be removed first. More concretely, if you are dividing traffic among tasks, you can
   * decrease the number of tasks from 15 and 10, killing off the final 5 tasks, and {@code
   * consistentHash} will handle it. If, however, you are dividing traffic among servers {@code
   * alpha}, {@code bravo}, and {@code charlie} and you occasionally need to take each of the
   * servers offline, {@code consistentHash} will be a poor fit: It provides no way for you to
   * specify which of the three buckets is disappearing. Thus, if your buckets change from {@code
   * [alpha, bravo, charlie]} to {@code [bravo, charlie]}, it will assign all the old {@code alpha}
   * traffic to {@code bravo} and all the old {@code bravo} traffic to {@code charlie}, rather than
   * letting {@code bravo} keep its traffic.
   * </ul>
   *
   *
   * <p>See the <a href="http://en.wikipedia.org/wiki/Consistent_hashing">Wikipedia article on
   * consistent hashing</a> for more information.
   */
  public static int consistentHash(HashCode hashCode, int buckets) {
    return consistentHash(hashCode.padToLong(), buckets);
  }

  /**
   * Assigns to {@code input} a "bucket" in the range {@code [0, buckets)}, in a uniform manner that
   * minimizes the need for remapping as {@code buckets} grows. That is, {@code consistentHash(h,
   * n)} equals:
   *
   * <ul>
   * <li>{@code n - 1}, with approximate probability {@code 1/n}
   * <li>{@code consistentHash(h, n - 1)}, otherwise (probability {@code 1 - 1/n})
   * </ul>
   *
   * <p>This method is suitable for the common use case of dividing work among buckets that meet the
   * following conditions:
   *
   * <ul>
   * <li>You want to assign the same fraction of inputs to each bucket.
   * <li>When you reduce the number of buckets, you can accept that the most recently added buckets
   * will be removed first. More concretely, if you are dividing traffic among tasks, you can
   * decrease the number of tasks from 15 and 10, killing off the final 5 tasks, and {@code
   * consistentHash} will handle it. If, however, you are dividing traffic among servers {@code
   * alpha}, {@code bravo}, and {@code charlie} and you occasionally need to take each of the
   * servers offline, {@code consistentHash} will be a poor fit: It provides no way for you to
   * specify which of the three buckets is disappearing. Thus, if your buckets change from {@code
   * [alpha, bravo, charlie]} to {@code [bravo, charlie]}, it will assign all the old {@code alpha}
   * traffic to {@code bravo} and all the old {@code bravo} traffic to {@code charlie}, rather than
   * letting {@code bravo} keep its traffic.
   * </ul>
   *
   *
   * <p>See the <a href="http://en.wikipedia.org/wiki/Consistent_hashing">Wikipedia article on
   * consistent hashing</a> for more information.
   */
  public static int consistentHash(long input, int buckets) {
    checkArgument(buckets > 0, "buckets must be positive: %s", buckets);
    LinearCongruentialGenerator generator = new LinearCongruentialGenerator(input);
    int candidate = 0;
    int next;

    // Jump from bucket to bucket until we go out of range
    while (true) {
      next = (int) ((candidate + 1) / generator.nextDouble());
      if (next >= 0 && next < buckets) {
        candidate = next;
      } else {
        return candidate;
      }
    }
  }

/**
   * Linear CongruentialGenerator to use for consistent hashing. See
   * http://en.wikipedia.org/wiki/Linear_congruential_generator
   */
  private static final class LinearCongruentialGenerator {
    private long state;

    public LinearCongruentialGenerator(long seed) {
      this.state = seed;
    }

    public double nextDouble() {
      state = 2862933555777941757L * state + 1;
      return ((double) ((int) (state >>> 33) + 1)) / (0x1.0p31);
    }
  }

使用实例

    @Test
    public void testJumpHash(){
        List<String> nodes = Arrays.asList("ins1","ins2","ins3","ins4");
        List<String> keys = Arrays.asList("key1","key2","key3","key4");
        keys.stream().forEach(e -> {
            int bucket = Hashing.consistentHash(Hashing.md5().hashString(e, Charsets.UTF_8), nodes.size());
            String node = nodes.get(bucket);
            System.out.println(e + " >> " + node);
        });
    }

doc

  • A Fast, Minimal Memory, Consistent Hash Algorithm
  • 一个速度快内存占用小的一致性哈希算法
  • jump Consistent hash:零内存消耗,均匀,快速,简洁,来自Google的一致性哈希算法

原文发布于微信公众号 - 码匠的流水账(geek_luandun)

原文发表时间:2017-11-11

本文参与腾讯云自媒体分享计划,欢迎正在阅读的你也加入,一起分享。

发表于

我来说两句

0 条评论
登录 后参与评论

相关文章

来自专栏函数式编程语言及工具

Akka(23): Stream:自定义流构件功能-Custom defined stream processing stages

    从总体上看:akka-stream是由数据源头Source,流通节点Flow和数据流终点Sink三个框架性的流构件(stream components)...

4308
来自专栏GIS讲堂

geotools获取给定点的DEM高程值

1、在web端绘制一条曲线; 2、获取各节点处的高程值; 3、根据高程值绘制高程堆积图。

1705
来自专栏编码小白

ofbiz 服务引擎(一) controller中服务的调用解析

首先根据handler-controller.xml文件中对应handler文件,然后运行RequestHandler中的runEvent方法,方法如下: /*...

3564
来自专栏码匠的流水账

聊聊storm的LoggingMetricsConsumer

storm-2.0.0/storm-client/src/jvm/org/apache/storm/metric/LoggingMetricsConsumer....

1333
来自专栏码匠的流水账

聊聊storm的LoggingMetricsConsumer

storm-2.0.0/storm-client/src/jvm/org/apache/storm/metric/LoggingMetricsConsumer....

1043
来自专栏后端之路

来自Google的瓜娃简介之Collections2

注:以下分析均基于Guava18 背景 除了我们会如此命名XXXService2 原来Google也会这样命名O(∩_∩)O哈! 由于jdk已经占据了Colle...

3055
来自专栏码匠的流水账

聊聊flink的RichParallelSourceFunction

flink-streaming-java_2.11-1.6.2-sources.jar!/org/apache/flink/streaming/api/func...

1372
来自专栏小樱的经验随笔

Codeforces 725B Food on the Plane

B. Food on the Plane time limit per test:2 seconds memory limit per test:256 meg...

3678
来自专栏函数式编程语言及工具

Akka(17): Stream:数据流基础组件-Source,Flow,Sink简介

    在大数据程序流行的今天,许多程序都面临着共同的难题:程序输入数据趋于无限大,抵达时间又不确定。一般的解决方法是采用回调函数(callback-funct...

3086
来自专栏张善友的专栏

如何结合IbatisNet的LIST遍历实现模糊查询

我仿照Java的Spring+Ibatis+Struct用Castle+IBatisNet+Asp.net的开发框架的DAO的基类:BaseSqlMapDao内...

2089

扫码关注云+社区