Flink Standalone HA搭建指南

从大数据到人工智能

发布于 2022-03-07 10:40:53

8070

发布于 2022-03-07 10:40:53

文章被收录于专栏：大数据-BigData

基本环境

集群主机	IP
host147	10.45.151.147
host148	10.45.151.148
host149	10.45.151.149

配置域名映射

vim /etc/hosts

10.45.151.147 host147
10.45.151.148 host148
10.45.151.149 host149

下载Flink安装包，将其放到/data目录下，并解压

mkdir /data && cd /data
wget http://192.168.1.3:11180/downloads/flink-1.13.5-bin-scala_2.12.tgz
tar zxvf flink-1.13.5-bin-scala_2.12.tgz
cd flink-1.13.5-bin

使用flink内置zk启动zk集群

选定上述三个节点作为安装zk集群的节点。

在conf/flink-conf.yaml中设置java信息，新增如下配置

java.home: xxx

新建目录

mkdir /data/flink-1.13.5/zookeeper

修改配置文件conf/zoo.cfg

# The number of milliseconds of each tick
tickTime=2000

# The number of ticks that the initial  synchronization phase can take
initLimit=10

# The number of ticks that can pass between  sending a request and getting an acknowledgement
syncLimit=5

# The directory where the snapshot is stored.
dataDir=/data/flink-1.13.5/zookeeper

# The port at which the clients will connect
clientPort=2181

# ZooKeeper quorum peers
server.1=host147:2888:3888
server.2=host148:2888:3888
server.3=host149:2888:3888

进入host147主机的/data/flink-1.13.5/zookeeper目录，新建文件myid，并填入1

进入host148主机的/data/flink-1.13.5/zookeeper目录，新建文件myid，并填入2

进入host149主机的/data/flink-1.13.5/zookeeper目录，新建文件myid，并填入3

启动zookeeper集群

在host147中执行：/data/flink-1.13.5/bin/zookeeper.sh start 1

在host148中执行：/data/flink-1.13.5/bin/zookeeper.sh start 2

在host149中执行：/data/flink-1.13.5/bin/zookeeper.sh start 3

至此zookeeper高可用集群搭建完成

启动Flink高可用集群

由于不想用hdfs，本文以minio对象存储作为checkpoints和高可用存储路径。可参考文章：分布式minio搭建指南了解如何搭建minio集群。

修改Flink配置conf/flink-conf.yaml

(下述配置每台主机都一样）

配置minio访问信息

s3.endpoint: http://minio:9000
s3.path.style.access: true
s3.access.key: xxxxxxxxxxxxx
s3.secret.key: xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx

配置高可用信息

high-availability: zookeeper
high-availability.storageDir: s3://flink/ha
high-availability.zookeeper.quorum: host147:2181,host148:2181,host149:2181
high-availability.zookeeper.path.root: /flink
high-availability.cluster-id: /cluster_one

设置checkpoint信息

state.checkpoints.dir: s3://flink/chk
state.savepoints.dir: s3://flink/chk
state.backend.incremental: true
state.backend: rocksdb
execution.checkpointing.interval: 15000ms

在host147节点conf/flink-conf.yaml配置文件中，修改

jobmanager.rpc.address：host147

在host148节点conf/flink-conf.yaml配置文件中，修改

jobmanager.rpc.address：host148

将opt/flink-s3-fs-hadoop-1.13.5.jar复制到lib/目录中

host147节点启动jobmanager