使用 k3s 搭建 cilium + istio 实验环境
使用 k3s 搭建 cilium + istio 实验环境
以下步骤运行在 oracle cloud ARM Ampere A1 实例上，实例配置为：4C24G，操作系统为：ubuntu 22.04 LTS
一、安装 k3s
使用以下命令启动 k3s 集群，不安装 cni， 同时设置 cluster cidr 和 service cidr 防止和本机地址冲突( –cluster-cidr=‘10.42.0.0/16’ –service-cidr=‘10.253.0.0/16’ )
1 2 3 4 5 6 | curl -sfL -sfL https://get.k3s.io | \ INSTALL_K3S_EXEC="--flannel-backend=none \ --disable-network-policy \ --disable-kube-proxy \ --prefer-bundled-bin \ --disable=traefik,servicelb,local-storage" sh -s - |
|---|
二、安装 Cilium
1. 操作系统准备
禁用外部路由管理
一些发行版需要配置不管理外部路由，例如 Ubuntu 22.04，详细信息请查看GitHub issue 18706，编辑 /etc/systemd/networkd.conf 添加以下设置：
1 2 3 | [Network] ManageForeignRoutes=no ManageForeignRoutingPolicyRules=no |
|---|
重启systemd-networkd服务
1 2 | systemctl daemon-reload systemctl restart systemd-networkd |
|---|
2. 安装 Cilium CLI
1 2 3 4 5 6 7 8 9 10 | CILIUM_CLI_VERSION=$(curl -s https://raw.githubusercontent.com/cilium/cilium-cli/main/stable.txt) CLI_ARCH=amd64 if [ "$(uname -m)" = "aarch64" ]; then CLI_ARCH=arm64; fi curl -L --fail --remote-name-all https://github.com/cilium/cilium-cli/releases/download/${CILIUM_CLI_VERSION}/cilium-linux-${CLI_ARCH}.tar.gz{,.sha256sum} sha256sum --check cilium-linux-${CLI_ARCH}.tar.gz.sha256sum sudo tar xzvfC cilium-linux-${CLI_ARCH}.tar.gz /usr/local/bin rm cilium-linux-${CLI_ARCH}.tar.gz{,.sha256sum} #查看当前安装版本 cilium version --client |
|---|
3. 安装 Cilium
同时安装 Hubble & Hubble UI
为确保 Cilium 不干扰 Istio，必须使用 –config bpf-lb-sock-hostns-only=true cilium CLI 标志或 socketLB.hostNamespaceOnly=true 来部署 Cilium。同时 ipam.operator.clusterPoolIPv4PodCIDRList 指定 k3s 默认的 pod CIDR。
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 | export KUBECONFIG=/etc/rancher/k3s/k3s.yaml API_SERVER_IP='10.0.0.214' # 服务器 ip 地址 API_SERVER_PORT=6443 cilium install --set hubble.relay.enabled=true \ --set hubble.ui.enabled=true \ --set operator.replicas=1 \ --set kubeProxyReplacement=strict \ --set ipam.operator.clusterPoolIPv4PodCIDRList="10.42.0.0/16" \ --set socketLB.hostNamespaceOnly=true \ --set k8sServiceHost=${API_SERVER_IP} \ --set k8sServicePort=${API_SERVER_PORT} \ --set tunnel=disabled \ --set autoDirectNodeRoutes=true \ --set ipv4NativeRoutingCIDR="10.42.0.0/16" \ --set bpf.masquerade=true --version 1.14.1 |
|---|
NATIVE ROUTING：
1 2 3 | --set tunnel=disabled\ --set autoDirectNodeRoutes=true\ --set ipv4NativeRoutingCIDR="10.42.0.0/16" |
|---|
IP 地址伪装(Masquerading):
1 | --set bpf.masquerade=true |
|---|
4. 验证 Cilium 安装
运行以下命令验证 cilium 时候正确安装
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 | # cilium status --wait /¯¯\ /¯¯\__/¯¯\ Cilium: OK \__/¯¯\__/ Operator: OK /¯¯\__/¯¯\ Envoy DaemonSet: disabled (using embedded mode) \__/¯¯\__/ Hubble Relay: OK \__/ ClusterMesh: disabled DaemonSet cilium Desired: 1, Ready: 1/1, Available: 1/1 Deployment hubble-relay Desired: 1, Ready: 1/1, Available: 1/1 Deployment hubble-ui Desired: 1, Ready: 1/1, Available: 1/1 Deployment cilium-operator Desired: 1, Ready: 1/1, Available: 1/1 Containers: cilium-operator Running: 1 cilium Running: 1 hubble-relay Running: 1 hubble-ui Running: 1 Cluster Pods: 6/6 managed by Cilium Helm chart version: 1.14.0 Image versions cilium quay.io/cilium/cilium:v1.14.0@sha256:5a94b561f4651fcfd85970a50bc78b201cfbd6e2ab1a03848eab25a82832653a: 1 hubble-relay quay.io/cilium/hubble-relay:v1.14.0@sha256:bfe6ef86a1c0f1c3e8b105735aa31db64bcea97dd4732db6d0448c55a3c8e70c: 1 hubble-ui quay.io/cilium/hubble-ui:v0.12.0@sha256:1c876cfa1d5e35bc91e1025c9314f922041592a88b03313c22c1f97a5d2ba88f: 1 hubble-ui quay.io/cilium/hubble-ui-backend:v0.12.0@sha256:8a79a1aad4fc9c2aa2b3e4379af0af872a89fcec9d99e117188190671c66fc2e: 1 cilium-operator quay.io/cilium/operator-generic:v1.14.0@sha256:3014d4bcb8352f0ddef90fa3b5eb1bbf179b91024813a90a0066eb4517ba93c9: 1 |
|---|
运行以下命令验证集群网络的连通性
1 2 3 4 5 6 | $ cilium connectivity test ℹ️ Single-node environment detected, enabling single-node connectivity test ℹ️ Monitor aggregation detected, will skip some flow validation steps ........ ✅ All 42 tests (184 actions) successful, 13 tests skipped, 0 scenarios skipped. |
|---|
三、安装 Istio
1. 下载 Istio
使用以下脚本地址下载 Istio
1 | curl -L https://istio.io/downloadIstio | sh - |
|---|
转到 Istio 包目录。例如，如果包是istio-1.18.2：
1 | $ cd istio-1.18.2 |
|---|
安装目录包含：
-
samples/目录下的示例应用程序 -
bin/目录下的[istioctl](https://istio.io/latest/zh/docs/reference/commands/istioctl)客户端二进制文件。
将istioctl客户端添加到路径：
1 | cp bin/istioctl /usr/local/bin/ |
|---|
2. 使用预设文件部署 istio
采用demo配置组合, 选择它是因为它包含了一组专为测试准备的功能集合，另外还有用于生产或性能测试的配置组合。
1 2 3 4 5 6 7 8 | # export KUBECONFIG=/etc/rancher/k3s/k3s.yaml # istioctl install --set profile=demo -y ✔ Istio core installed ✔ Istiod installed ✔ Egress gateways installed ✔ Ingress gateways installed ✔ Installation complete Making this installation the default for injection and validation |
|---|
命名空间添加标签，指示 Istio 在部署应用的时候，自动注入 Envoy 边车代理
1 2 | $ kubectl label namespace default istio-injection=enabled namespace/default labeled |
|---|
四、卸载
当 k3s 环境出现问题，或者需要重建，需要使用以下方法：
1 2 3 4 5 6 7 8 9 10 | ip link delete cilium_host ip link delete cilium_net ip link delete cilium_vxlan iptables-save | grep -iv cilium | iptables-restore ip6tables-save | grep -iv cilium | ip6tables-restore /usr/local/bin/k3s-killall.sh /usr/local/bin/k3s-uninstall.sh |
|---|
至此完成 cilium + Istio 实验环境搭建，后续将进行 cilium 和 Istio 相关的实验。