使用Kubernetes中的StatefulSet对象查找没有固定IP的kafka broker时出错
使用Kubernetes中的StatefulSet对象查找没有固定IP的kafka broker时出错
提问于 2020-09-30 18:42:25
我正在尝试使用confluentinc docker镜像在Kubernetes上部署Zookeeper和Kafka。我的解决方案基于这个question和这个post。Zookeeper正在运行，日志中没有错误。我想部署3卡夫卡经纪人使用StatefulSet。我的yaml文件的问题是，当使用3个代理时，我不知道如何为Kafka配置KAFKA_ADVERTISED_LISTENERS属性。
以下是zookeeper的yaml文件：
apiVersion: v1
kind: Service
metadata:
name: zookeeper
labels:
app: zookeeper
spec:
clusterIP: None
ports:
- name: client
port: 2181
protocol: TCP
- name: follower
port: 2888
protocol: TCP
- name: leader
port: 3888
protocol: TCP
selector:
app: zookeeper
---
apiVersion: apps/v1
kind: StatefulSet
metadata:
name: zookeeper
spec:
replicas: 1
serviceName: zookeeper
selector:
matchLabels:
app: zookeeper # has to match .spec.template.metadata.labels
template:
metadata:
labels:
app: zookeeper # has to match .spec.selector.matchLabels
spec:
hostname: zookeeper
containers:
- name: zookeeper
image: confluentinc/cp-zookeeper:5.5.0
ports:
- containerPort: 2181
env:
- name: ZOOKEEPER_CLIENT_PORT
value: "2181"
- name: ZOOKEEPER_ID
value: "1"
- name: ZOOKEEPER_SERVER_1
value: zookeeper对于kafka经纪人：
apiVersion: v1
kind: Service
metadata:
name: kafka-service
labels:
app: kafka
spec:
type: LoadBalancer
ports:
- port: 9092
name: kafka-port
protocol: TCP
selector:
app: kafka
id: "0"
---
apiVersion: apps/v1
kind: StatefulSet
metadata:
name: kafka
spec:
replicas: 3
serviceName: kafka
podManagementPolicy: OrderedReady
selector:
matchLabels:
app: kafka # has to match .spec.template.metadata.labels
template:
metadata:
labels:
app: kafka # has to match .spec.selector.matchLabels
spec:
containers:
- name: kafka
image: confluentinc/cp-enterprise-kafka:5.5.0
ports:
- containerPort: 9092
env:
- name: KAFKA_ZOOKEEPER_CONNECT
value: zookeeper:2181 # zookeeper-2.zookeeper.default.svc.cluster.local
- name: KAFKA_ADVERTISED_LISTENERS
value: "LISTENER_0://kafka-0:9092,LISTENER_1://kafka-1:9093,LISTENER_2://kafka-2:9094"
- name: KAFKA_LISTENER_SECURITY_PROTOCOL_MAP
value: "LISTENER_0:PLAINTEXT,LISTENER_1:PLAINTEXT,LISTENER_2:PLAINTEXT"
- name: KAFKA_INTER_BROKER_LISTENER_NAME
value: LISTENER_0我运行了3个kafka pod，kafka-0正在连接，但kafka-1和kafka-2没有连接。
$ kubectl get pods -o wide
NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES
kafka-0 1/1 Running 0 4m12s 172.17.0.4 minikube <none> <none>
kafka-1 1/1 Running 5 4m9s 172.17.0.5 minikube <none> <none>
kafka-2 0/1 CrashLoopBackOff 4 4m7s 172.17.0.6 minikube <none> <none>
zookeeper-0 1/1 Running 0 21m 172.17.0.3 minikube <none> <none>错误是说我已经在第一个pod kafka-0中通告了kafka-0:9092,kafka-1:9093,kafka-2:9094。所以，我认为它必须是动态的。如何配置？
[2020-09-30 14:56:40,519] ERROR [KafkaServer id=1017] Fatal error during KafkaServer startup. Prepare to shutdown (kafka.server.KafkaServer)
java.lang.IllegalArgumentException: requirement failed: Configured end points kafka-0:9092,kafka-1:9093,kafka-2:9094 in advertised listeners are already registered by broker 1012
at kafka.server.KafkaServer.$anonfun$createBrokerInfo$3(KafkaServer.scala:436)
at kafka.server.KafkaServer.$anonfun$createBrokerInfo$3$adapted(KafkaServer.scala:434)
at scala.collection.mutable.ResizableArray.foreach(ResizableArray.scala:62)
at scala.collection.mutable.ResizableArray.foreach$(ResizableArray.scala:55)
at scala.collection.mutable.ArrayBuffer.foreach(ArrayBuffer.scala:49)
at kafka.server.KafkaServer.createBrokerInfo(KafkaServer.scala:434)
at kafka.server.KafkaServer.startup(KafkaServer.scala:293)
at io.confluent.support.metrics.SupportedServerStartable.startup(SupportedServerStartable.java:114)
at io.confluent.support.metrics.SupportedKafka.main(SupportedKafka.java:66)回答 1
Stack Overflow用户
回答已采纳
发布于 2020-10-01 18:48:23
我一直在阅读这篇博文"Kafka Listeners - Explained"，我能够使用以下配置配置3个Kafka代理。
apiVersion: apps/v1
kind: StatefulSet
metadata:
name: kafka
spec:
replicas: 3
serviceName: kafka
podManagementPolicy: OrderedReady
selector:
matchLabels:
app: kafka # has to match .spec.template.metadata.labels
template:
metadata:
labels:
app: kafka # has to match .spec.selector.matchLabels
spec:
restartPolicy: Always
containers:
- name: kafka
image: confluentinc/cp-enterprise-kafka:5.5.0
resources:
requests:
memory: "256Mi"
cpu: "250m"
limits: # limit of 0.5 cpu and 512MiB of memory
memory: "512Mi"
cpu: "500m"
# imagePullPolicy: Always
ports:
- containerPort: 9092
name: kafka-0
- containerPort: 9093
name: kafka-1
- containerPort: 9094
name: kafka-2
env:
- name: MY_METADATA_NAME
valueFrom:
fieldRef:
fieldPath: metadata.name
- name: MY_POD_IP
valueFrom:
fieldRef:
fieldPath: status.podIP
- name: STAS_DELAY
value: "120"
- name: KAFKA_ZOOKEEPER_CONNECT
value: zookeeper:2181 # zookeeper-2.zookeeper.default.svc.cluster.local
- name: KAFKA_ADVERTISED_LISTENERS
value: "INSIDE://$(MY_POD_IP):9092"
- name: KAFKA_LISTENERS
value: "INSIDE://$(MY_POD_IP):9092"
- name: KAFKA_LISTENER_SECURITY_PROTOCOL_MAP
value: "INSIDE:PLAINTEXT"
- name: KAFKA_INTER_BROKER_LISTENER_NAME
value: "INSIDE"页面原文内容由Stack Overflow提供。腾讯云小微IT领域专用引擎提供翻译支持
原文链接：
https://stackoverflow.com/questions/64135892
复制相关文章
相似问题
腾讯云开发者
