首页
学习
活动
专区
工具
TVP
发布
社区首页 >专栏 >聊聊flink的AscendingTimestampExtractor

聊聊flink的AscendingTimestampExtractor

作者头像
code4it
发布2018-12-28 12:31:55
4350
发布2018-12-28 12:31:55
举报
文章被收录于专栏:码匠的流水账码匠的流水账

本文主要研究一下flink的AscendingTimestampExtractor

AscendingTimestampExtractor

flink-streaming-java_2.11-1.7.0-sources.jar!/org/apache/flink/streaming/api/functions/timestamps/AscendingTimestampExtractor.java

/**
 * A timestamp assigner and watermark generator for streams where timestamps are monotonously
 * ascending. In this case, the local watermarks for the streams are easy to generate, because
 * they strictly follow the timestamps.
 *
 * @param <T> The type of the elements that this function can extract timestamps from
 */
@PublicEvolving
public abstract class AscendingTimestampExtractor<T> implements AssignerWithPeriodicWatermarks<T> {

    private static final long serialVersionUID = 1L;

    /** The current timestamp. */
    private long currentTimestamp = Long.MIN_VALUE;

    /** Handler that is called when timestamp monotony is violated. */
    private MonotonyViolationHandler violationHandler = new LoggingHandler();


    /**
     * Extracts the timestamp from the given element. The timestamp must be monotonically increasing.
     *
     * @param element The element that the timestamp is extracted from.
     * @return The new timestamp.
     */
    public abstract long extractAscendingTimestamp(T element);

    /**
     * Sets the handler for violations to the ascending timestamp order.
     *
     * @param handler The violation handler to use.
     * @return This extractor.
     */
    public AscendingTimestampExtractor<T> withViolationHandler(MonotonyViolationHandler handler) {
        this.violationHandler = requireNonNull(handler);
        return this;
    }

    // ------------------------------------------------------------------------

    @Override
    public final long extractTimestamp(T element, long elementPrevTimestamp) {
        final long newTimestamp = extractAscendingTimestamp(element);
        if (newTimestamp >= this.currentTimestamp) {
            this.currentTimestamp = newTimestamp;
            return newTimestamp;
        } else {
            violationHandler.handleViolation(newTimestamp, this.currentTimestamp);
            return newTimestamp;
        }
    }

    @Override
    public final Watermark getCurrentWatermark() {
        return new Watermark(currentTimestamp == Long.MIN_VALUE ? Long.MIN_VALUE : currentTimestamp - 1);
    }

    //......
}
  • AscendingTimestampExtractor抽象类实现AssignerWithPeriodicWatermarks接口的extractTimestamp及getCurrentWatermark方法,同时声明抽象方法extractAscendingTimestamp供子类实现
  • AscendingTimestampExtractor适用于elements的时间在每个parallel task里头是单调递增(timestamp monotony)的场景,extractTimestamp这里先是调用子类实现的extractAscendingTimestamp方法从element提取newTimestamp,然后返回,对于违反timestamp monotony的,这里调用MonotonyViolationHandler进行处理
  • getCurrentWatermark方法在currentTimestamp不为Long.MIN_VALUE时返回Watermark(currentTimestamp - 1)

MonotonyViolationHandler

flink-streaming-java_2.11-1.7.0-sources.jar!/org/apache/flink/streaming/api/functions/timestamps/AscendingTimestampExtractor.java

    /**
     * Interface for handlers that handle violations of the monotonous ascending timestamps
     * property.
     */
    public interface MonotonyViolationHandler extends java.io.Serializable {

        /**
         * Called when the property of monotonously ascending timestamps is violated, i.e.,
         * when {@code elementTimestamp < lastTimestamp}.
         *
         * @param elementTimestamp The timestamp of the current element.
         * @param lastTimestamp The last timestamp.
         */
        void handleViolation(long elementTimestamp, long lastTimestamp);
    }

    /**
     * Handler that does nothing when timestamp monotony is violated.
     */
    public static final class IgnoringHandler implements MonotonyViolationHandler {
        private static final long serialVersionUID = 1L;

        @Override
        public void handleViolation(long elementTimestamp, long lastTimestamp) {}
    }

    /**
     * Handler that fails the program when timestamp monotony is violated.
     */
    public static final class FailingHandler implements MonotonyViolationHandler {
        private static final long serialVersionUID = 1L;

        @Override
        public void handleViolation(long elementTimestamp, long lastTimestamp) {
            throw new RuntimeException("Ascending timestamps condition violated. Element timestamp "
                    + elementTimestamp + " is smaller than last timestamp " + lastTimestamp);
        }
    }

    /**
     * Handler that only logs violations of timestamp monotony, on WARN log level.
     */
    public static final class LoggingHandler implements MonotonyViolationHandler {
        private static final long serialVersionUID = 1L;

        private static final Logger LOG = LoggerFactory.getLogger(AscendingTimestampExtractor.class);

        @Override
        public void handleViolation(long elementTimestamp, long lastTimestamp) {
            LOG.warn("Timestamp monotony violated: {} < {}", elementTimestamp, lastTimestamp);
        }
    }
  • MonotonyViolationHandler继承了Serializable,它定义了handleViolation方法,这个接口内置有三个实现类,分别是IgnoringHandler、FailingHandler、FailingHandler
  • IgnoringHandler的handleViolation方法不做任何处理;FailingHandler的handleViolation会抛出RuntimeException;LoggingHandler的handleViolation方法会打印warn日志
  • AscendingTimestampExtractor默认使用的是LoggingHandler,也可以通过withViolationHandler方法来进行设置

实例

    @Test
    public void testWithFailingHandler() {
        AscendingTimestampExtractor<Long> extractor = (new AscendingTimestampExtractorTest.LongExtractor()).withViolationHandler(new FailingHandler());
        this.runValidTests(extractor);

        try {
            this.runInvalidTest(extractor);
            Assert.fail("should fail with an exception");
        } catch (Exception var3) {
            ;
        }

    }

    private void runValidTests(AscendingTimestampExtractor<Long> extractor) {
        Assert.assertEquals(13L, extractor.extractTimestamp(13L, -1L));
        Assert.assertEquals(13L, extractor.extractTimestamp(13L, 0L));
        Assert.assertEquals(14L, extractor.extractTimestamp(14L, 0L));
        Assert.assertEquals(20L, extractor.extractTimestamp(20L, 0L));
        Assert.assertEquals(20L, extractor.extractTimestamp(20L, 0L));
        Assert.assertEquals(20L, extractor.extractTimestamp(20L, 0L));
        Assert.assertEquals(500L, extractor.extractTimestamp(500L, 0L));
        Assert.assertEquals(9223372036854775806L, extractor.extractTimestamp(9223372036854775806L, 99999L));
    }

    private void runInvalidTest(AscendingTimestampExtractor<Long> extractor) {
        Assert.assertEquals(1000L, extractor.extractTimestamp(1000L, 100L));
        Assert.assertEquals(1000L, extractor.extractTimestamp(1000L, 100L));
        Assert.assertEquals(999L, extractor.extractTimestamp(999L, 100L));
    }

    private static class LongExtractor extends AscendingTimestampExtractor<Long> {
        private static final long serialVersionUID = 1L;

        private LongExtractor() {
        }

        public long extractAscendingTimestamp(Long element) {
            return element;
        }
    }
  • 这里使用withViolationHandler设置了violationHandler为FailingHandler,在遇到999这个时间的时候,由于比之前的1000小,因而会调用MonotonyViolationHandler.handleViolation方法

小结

  • flink为了方便开发提供了几个内置的Pre-defined Timestamp Extractors / Watermark Emitters,其中一个就是AscendingTimestampExtractor
  • AscendingTimestampExtractor抽象类实现AssignerWithPeriodicWatermarks接口的extractTimestamp及getCurrentWatermark方法,同时声明抽象方法extractAscendingTimestamp供子类实现
  • AscendingTimestampExtractor适用于elements的时间在每个parallel task里头是单调递增的,对于违反timestamp monotony的,这里调用MonotonyViolationHandler的handleViolation方法进行处理;MonotonyViolationHandler继承了Serializable,它定义了handleViolation方法,这个接口内置有三个实现类,分别是IgnoringHandler、FailingHandler、FailingHandler

doc

  • Pre-defined Timestamp Extractors / Watermark Emitters
本文参与 腾讯云自媒体分享计划,分享自微信公众号。
原始发表:2018-12-20,如有侵权请联系 cloudcommunity@tencent.com 删除

本文分享自 码匠的流水账 微信公众号,前往查看

如有侵权,请联系 cloudcommunity@tencent.com 删除。

本文参与 腾讯云自媒体分享计划  ,欢迎热爱写作的你一起参与!

评论
登录后参与评论
0 条评论
热度
最新
推荐阅读
目录
  • AscendingTimestampExtractor
  • MonotonyViolationHandler
  • 实例
  • 小结
  • doc
领券
问题归档专栏文章快讯文章归档关键词归档开发者手册归档开发者手册 Section 归档