k8s运维：记 istio ingressgateway 关于TCP路由的一个小坑
k8s运维：记 istio ingressgateway 关于TCP路由的一个小坑
机械视角
发布于 2020-12-31 10:45:20
发布于 2020-12-31 10:45:20
因为之前都是用 istio ingressgateway 做 L7 负载，这次收到一个集群 kafka broker 暴露需求，在设置的时候习惯了 HTTP 的思路，所以才了坑。
首先来说下 istio ingressgateway 做 L7 层负载工作原理。
首先我们会先设置一个 gateway 绑定作用的 ingressgateway：
apiVersion: networking.istio.io/v1alpha3
kind: Gateway
metadata:
name: external-test-gateway
spec:
selector:
istio: ingressgateway-external
servers:
- hosts:
- '*.rcmd.testing.mpengine'
port:
name: http
number: 16354
protocol: HTTPgateway 资源会被 istiod 监控，当一个新的 gateway 创建的时候， istiod 会根据 gateway 的 selector 选择符合条件的 istio proxyv2 的 pod (也就是istio ingressgateway的运行实例)，我们可以看 istiod 日志：
$ kubectl logs -f istiod-xxxxx -nistio-system
2020-12-30T09:40:57.178237Z info ads CDS: PUSH for node:istio-ingressgateway-external-5b4765b84-vrmwq.istio-system resources:17
2020-12-30T09:40:57.178307Z info ads EDS: PUSH for node:istio-ingressgateway-external-5b4765b84-vrmwq.istio-system resources:16 empty:0 cached:16/16
2020-12-30T09:40:57.178581Z info ads LDS: PUSH for node:istio-ingressgateway-external-5b4765b84-vrmwq.istio-system resources:4
2020-12-30T09:40:57.178896Z info ads RDS: PUSH for node:istio-ingressgateway-external-5b4765b84-vrmwq.istio-system resources:2
2020-12-30T09:41:13.584044Z info ads Push debounce stable[5] 1: 100.121839ms since last change, 100.121637ms since last push, full=true
2020-12-30T09:41:13.584239Z info ads XDS: Pushing:2020-12-30T09:41:13Z/1 Services:63 ConnectedEndpoints:2
2020-12-30T09:41:13.584569Z info ads CDS: PUSH for node:istio-ingressgateway-85b5cf599b-qcj85.istio-system resources:1
2020-12-30T09:41:13.584615Z info ads LDS: PUSH for node:istio-ingressgateway-85b5cf599b-qcj85.istio-system resources:0
2020-12-30T09:41:13.585106Z info ads CDS: PUSH for node:istio-ingressgateway-external-5b4765b84-vrmwq.istio-system resources:17
2020-12-30T09:41:13.585248Z warn buildGatewayListeners: found 0 services on port 16354: map[]
2020-12-30T09:41:13.585466Z info ads LDS: PUSH for node:istio-ingressgateway-external-5b4765b84-vrmwq.istio-system resources:5
2020-12-30T09:41:13.585674Z info ads RDS: PUSH for node:istio-ingressgateway-external-5b4765b84-vrmwq.istio-system resources:2
2020-12-30T09:41:13.595165Z warn constructed http route config for route http.16354 on port 16354 with no vhosts; Setting up a default 404 vhost
2020-12-30T09:41:13.595493Z info ads RDS: PUSH for node:istio-ingressgateway-external-5b4765b84-vrmwq.istio-system resources:3对于 HTTP 协议的 gateway 会直接在 istio-ingressgateway 上开启监听端口：
$ kubectl exec -it -nistio-system istio-ingressgateway-external-xxxxxx -- bash
$ netstat -anp | grep 16354
tcp 0 0 0.0.0.0:16354 0.0.0.0:* LISTEN 19/envoy这时候再设置 VirtualService 让 envoy 匹配转发到对应服务，如下：
apiVersion: networking.istio.io/v1beta1
kind: VirtualService
metadata:
name: test-backend
spec:
gateways:
- external-test-gateway
hosts:
- test.rcmd.testing.mpengine
http:
- route:
- destination:
host: prometheus.cluster-monitor.svc.cluster.local
port:
number: 80这是 HTTP 协议转发实现，但istio ingressgateway 在 TCP 协议是不同的，对于 TCP 类型的 gateway 只有同时修改 gateway 和 virtualService才能开启端口，同样做个测试：
apiVersion: networking.istio.io/v1alpha3
kind: Gateway
metadata:
name: external-test-gateway
namespace: rcmd
spec:
selector:
istio: ingressgateway-external
servers:
- hosts:
- '*.rcmd.testing.mpengine'
port:
name: tcp
number: 16354
protocol: TCP # 改成TCP协议接口到 ingressgateway 的 pod 里面可以发现 16354 端口并没有启动：
$ kubectl exec -it -nistio-system istio-ingressgateway-external-xxxxxx -- bash
$ netstat -anp | grep 16354 # 为空看 istiod 日志：
020-12-30T10:14:30.345439Z info ads Push debounce stable[23] 1: 100.094272ms since last change, 100.094082ms since last push, full=true
2020-12-30T10:14:30.345683Z info ads XDS: Pushing:2020-12-30T10:14:30Z/11 Services:63 ConnectedEndpoints:2
2020-12-30T10:14:30.346030Z info ads CDS: PUSH for node:istio-ingressgateway-85b5cf599b-qcj85.istio-system resources:1
2020-12-30T10:14:30.346065Z info ads LDS: PUSH for node:istio-ingressgateway-85b5cf599b-qcj85.istio-system resources:0
2020-12-30T10:14:30.346784Z info ads CDS: PUSH for node:istio-ingressgateway-external-5b4765b84-v6bxd.istio-system resources:17
2020-12-30T10:14:30.346816Z warn buildGatewayListeners: found 0 services on port 16354: map[]
2020-12-30T10:14:30.346823Z warn no virtual service bound to gateway: rcmd/external-test-gateway
2020-12-30T10:14:30.346941Z info gateway omitting listener "0.0.0.0_16354" due to: must have more than 0 chains in listener "0.0.0.0_16354"
2020-12-30T10:14:30.347122Z info ads LDS: PUSH for node:istio-ingressgateway-external-5b4765b84-v6bxd.istio-system resources:4
2020-12-30T10:14:30.347295Z info ads RDS: PUSH for node:istio-ingressgateway-external-5b4765b84-v6bxd.istio-system resources:2添加一个 vs：
apiVersion: networking.istio.io/v1beta1
kind: VirtualService
metadata:
name: external-test-router
namespace: rcmd
spec:
gateways:
- external-test-gateway
hosts:
- "kafka-broker0.rcmd.testing.mpengine"
tcp:
- match:
- port: 16354
route:
- destination:
host: kafka-broker0.rcmd.svc.cluster.local
port:
number: 9092提交后，查看 istiod 日志，可以看到端口被成功创建 :
2020-12-30T10:21:11.541307Z info ads XDS: Pushing:2020-12-30T10:21:11Z/14 Services:63 ConnectedEndpoints:2
2020-12-30T10:21:11.541573Z info ads CDS: PUSH for node:istio-ingressgateway-85b5cf599b-qcj85.istio-system resources:1
2020-12-30T10:21:11.541621Z info ads LDS: PUSH for node:istio-ingressgateway-85b5cf599b-qcj85.istio-system resources:0
2020-12-30T10:21:11.542446Z info ads CDS: PUSH for node:istio-ingressgateway-external-5b4765b84-v6bxd.istio-system resources:17
2020-12-30T10:21:11.542602Z warn buildGatewayListeners: found 0 services on port 16354: map[]
2020-12-30T10:21:11.542767Z info ads LDS: PUSH for node:istio-ingressgateway-external-5b4765b84-v6bxd.istio-system resources:5
2020-12-30T10:21:11.543005Z info ads RDS: PUSH for node:istio-ingressgateway-external-5b4765b84-v6bxd.istio-system resources:2本文参与 腾讯云自媒体同步曝光计划，分享自作者个人站点/博客。
原始发表：2020-11-20，如有侵权请联系 cloudcommunity@tencent.com 删除
评论
登录后参与评论
推荐阅读