远程访问运行在kubernetes中的Kafka
远程访问运行在kubernetes中的Kafka
提问于 2019-09-25 08:01:45
我有一个节点Kafka broker运行在单个节点kubernetes环境中的一个吊舱内。我用这张图片为卡夫卡：https://hub.docker.com/r/wurstmeister/kafka
kafka版本= 1.1.0
Kubernetes集群在服务器上的VM中运行。VM在活动接口ens32 - 192.168.3.102上有以下IP
Kafka.yaml
apiVersion: extensions/v1beta1
kind: Deployment
metadata:
namespace: casb-deployment
name: kafkaservice
spec:
replicas: 1
template:
metadata:
labels:
app: kafkaservice
spec:
hostname: kafkaservice
containers:
- name: kafkaservice
imagePullPolicy: IfNotPresent
image: wurstmeister/kafka:1.1.0
env:
- name: KAFKA_BROKER_ID
value: "1"
# - name: KAFKA_ADVERTISED_HOST_NAME
# value: "kafkaservice"
- name: KAFKA_LISTENER_SECURITY_PROTOCOL_MAP
value: "INTERNAL_PLAINTEXT:PLAINTEXT,EXTERNAL_PLAINTEXT:PLAINTEXT"
- name: KAFKA_LISTENERS
value: "INTERNAL_PLAINTEXT://0.0.0.0:9092,EXTERNAL_PLAINTEXT://0.0.0.0:9093"
- name: KAFKA_ADVERTISED_LISTENERS
value: "INTERNAL_PLAINTEXT://kafkaservice:9092,EXTERNAL_PLAINTEXT://192.168.3.102:9093"
- name: KAFKA_INTER_BROKER_LISTENER_NAME
value: "INTERNAL_PLAINTEXT"
- name: KAFKA_CREATE_TOPICS
value: "topic-1:100:1,topic-2:1:1"
- name: KAFKA_ZOOKEEPER_CONNECT
value: "zookeeper:2181"
ports:
- name: port9092
containerPort: 9092
- name: port9093
containerPort: 9093
volumeMounts:
- mountPath: /kafka/kafka-logs-kafkaservice
name: kafka-volume
volumes:
- name: kafka-volume
hostPath:
path: /home/volume/kafka-logs
---
apiVersion: v1
kind: Service
metadata:
namespace: casb-deployment
name: kafkaservice
labels:
app: kafkaservice
spec:
selector:
app: kafkaservice
ports:
- name: port9092
port: 9092
targetPort: 9092
protocol: TCP
---
apiVersion: v1
kind: Service
metadata:
namespace: casb-deployment
name: kafkaservice-external
labels:
app: kafkaservice-external
spec:
selector:
app: kafkaservice
ports:
- name: port9093
port: 9093
protocol: TCP
nodePort: 30035
type: NodePort我能够从本地机器ping 192.168.3.102平VM，即kubernetes节点，并使用nodePort公开服务。
我还可以告诉telnet 192.168.3.102 30035，它提供：
Trying 192.168.3.102...
Connected to 192.168.3.102.
Escape character is '^]'.我试着在本地机器上运行kafka控制台消费者和制片人：
使用者：bin/kafka-控制台-使用者.bin引导-服务器192.168.3.102:30035 -主题-1-从开始
输出：
[2019-09-25 12:30:40,716] WARN [Consumer clientId=consumer-1, groupId=console-consumer-20551] Connection to node 1 could not be established. Broker may not be available. (org.apache.kafka.clients.NetworkClient)制片人：
bin/kafka-console-producer.sh --broker-list 192.168.3.102:30035 --topic topic-1输出：
[2019-09-25 12:32:07,958] WARN [Producer clientId=console-producer] Connection to node 1 could not be established. Broker may not be available. (org.apache.kafka.clients.NetworkClient)Netstat:
netstat -tulpn | grep 30035
tcp6 0 0 :::30035 :::* LISTEN 113545/kube-proxy我尝试运行一个基于python的使用者，即kafka-python=1.4.2，它给了我以下日志：
[2019-09-25T12:15:39+0500] INFO kafka.client Bootstrapping cluster metadata from [('192.168.3.102', 30035, <AddressFamily.AF_INET: 2>)]
[2019-09-25T12:15:39+0500] INFO kafka.conn <BrokerConnection node_id=bootstrap host=192.168.3.102:30035 <connecting> [IPv4 ('192.168.3.102', 30035)]>: connecting to 192.168.3.102:30035 [('192.168.3.102', 30035) IPv4]
[2019-09-25T12:15:39+0500] INFO kafka.conn <BrokerConnection node_id=bootstrap host=192.168.3.102:30035 <connecting> [IPv4 ('192.168.3.102', 30035)]>: Connection complete.
[2019-09-25T12:15:39+0500] INFO kafka.client Bootstrap succeeded: found 1 brokers and 26 topics.
[2019-09-25T12:15:39+0500] INFO kafka.conn <BrokerConnection node_id=bootstrap host=192.168.3.102:30035 <connected> [IPv4 ('192.168.3.102', 30035)]>: Closing connection.
[2019-09-25T12:15:39+0500] INFO kafka.conn <BrokerConnection node_id=1 host=192.168.3.102:9093 <connecting> [IPv4 ('192.168.3.102', 9093)]>: connecting to 192.168.3.102:9093 [('192.168.3.102', 9093) IPv4]
[2019-09-25T12:15:39+0500] INFO kafka.conn Probing node 1 broker version
[2019-09-25T12:15:39+0500] ERROR kafka.conn Connect attempt to <BrokerConnection node_id=1 host=192.168.3.102:9093 <connecting> [IPv4 ('192.168.3.102', 9093)]> returned error 111. Disconnecting.
[2019-09-25T12:15:39+0500] INFO kafka.conn <BrokerConnection node_id=1 host=192.168.3.102:9093 <connecting> [IPv4 ('192.168.3.102', 9093)]>: Closing connection. ConnectionError: 111 ECONNREFUSED
[2019-09-25T12:15:40+0500] INFO kafka.conn <BrokerConnection node_id=1 host=192.168.3.102:9093 <connecting> [IPv4 ('192.168.3.102', 9093)]>: connecting to 192.168.3.102:9093 [('192.168.3.102', 9093) IPv4]
[2019-09-25T12:15:40+0500] ERROR kafka.conn Connect attempt to <BrokerConnection node_id=1 host=192.168.3.102:9093 <connecting> [IPv4 ('192.168.3.102', 9093)]> returned error 111. Disconnecting.
[2019-09-25T12:15:40+0500] INFO kafka.conn <BrokerConnection node_id=1 host=192.168.3.102:9093 <connecting> [IPv4 ('192.168.3.102', 9093)]>: Closing connection. ConnectionError: 111 ECONNREFUSED
[2019-09-25T12:15:40+0500] INFO Activity URL collector Exception in activity url collector: NoBrokersAvailable从日志上看，连接似乎是建立的。
<connecting> [IPv4 ('192.168.3.102', 30035)]>: Connection complete.
Bootstrap succeeded: found 1 brokers and 26 topics.但后来被切断了。
请帮助我找出我错过了什么，我如何解决这个问题。谢谢。
Stack Overflow用户
发布于 2019-11-22 20:57:00
AWS/DigitalOcean/GCE/Azure中Kubernetes上Kafka/Zoo牧民的简单配置(外部访问)：
您可以通过常规二进制协议从AWS/DO/GCE外部连接到Kafka。连接是明文或SASL_PLAINTEXT (用户名/密码)。
卡夫卡集群是StatefulSet，因此您可以轻松地缩放集群。
页面原文内容由Stack Overflow提供。腾讯云小微IT领域专用引擎提供翻译支持
原文链接：
https://stackoverflow.com/questions/58093857
复制相关文章
相似问题
腾讯云开发者
