混沌工程之ChaosBlade-Operator使用之一模拟POD丢包场景
混沌工程之ChaosBlade-Operator使用之一模拟POD丢包场景
高楼Zee
发布于 2021-07-14 11:16:29
发布于 2021-07-14 11:16:29
文章被收录于专栏：7DGroup
ChaosBalde-Operator实现说明
ChaosBlade-Operator是ChaosBlade的K8s平台场景的实现。将混沌实验通过 Kubernetes 标准的 CRD 方式定义，用户可以像定义 Deployment 或 StatefulSet 那样定义 ChaosBlade 实验，只要对 kubectl 和 Kubernetes 对象有所了解，就可以轻松的创建、更新和删除实验场景；同时也可以通过 chaosblade cli 工具来操作实验场景。
ChaosBalde-Operator安装
使用 Helm 3 安装：
# 下载安装包
[root@s5 k8s]# wget -qO chaosblade-operator-1.2.0-v3.tgz https://chaosblade.oss-cn-hangzhou.aliyuncs.com/agent/github/1.2.0/chaosblade-operator-1.2.0-v3.tgz
# 为 chaosblade 创建一个 namespace
[root@s5 k8s]# kubectl create namespace chaosblade
namespace/chaosblade created
# 安装 ChaosBlade-Operator
[root@s5 k8s]# ./helm-darwin-amd64/helm install chaos chaosblade-operator-1.2.0-v3.tgz --set webhook.enable=true --namespace=chaosblade
W0621 14:39:16.362347 42437 warnings.go:70] apiextensions.k8s.io/v1beta1 CustomResourceDefinition is deprecated in v1.16+, unavailable in v1.22+; use apiextensions.k8s.io/v1 CustomResourceDefinition
W0621 14:39:16.375507 42437 warnings.go:70] apiextensions.k8s.io/v1beta1 CustomResourceDefinition is deprecated in v1.16+, unavailable in v1.22+; use apiextensions.k8s.io/v1 CustomResourceDefinition
W0621 14:39:18.394761 42437 warnings.go:70] apiextensions.k8s.io/v1beta1 CustomResourceDefinition is deprecated in v1.16+, unavailable in v1.22+; use apiextensions.k8s.io/v1 CustomResourceDefinition
W0621 14:39:20.669546 42437 warnings.go:70] rbac.authorization.k8s.io/v1beta1 ClusterRole is deprecated in v1.17+, unavailable in v1.22+; use rbac.authorization.k8s.io/v1 ClusterRole
W0621 14:39:20.674105 42437 warnings.go:70] rbac.authorization.k8s.io/v1beta1 ClusterRoleBinding is deprecated in v1.17+, unavailable in v1.22+; use rbac.authorization.k8s.io/v1 ClusterRoleBinding
W0621 14:39:20.687832 42437 warnings.go:70] admissionregistration.k8s.io/v1beta1 MutatingWebhookConfiguration is deprecated in v1.16+, unavailable in v1.22+; use admissionregistration.k8s.io/v1 MutatingWebhookConfiguration
W0621 14:39:20.734308 42437 warnings.go:70] rbac.authorization.k8s.io/v1beta1 ClusterRole is deprecated in v1.17+, unavailable in v1.22+; use rbac.authorization.k8s.io/v1 ClusterRole
W0621 14:39:20.742406 42437 warnings.go:70] rbac.authorization.k8s.io/v1beta1 ClusterRoleBinding is deprecated in v1.17+, unavailable in v1.22+; use rbac.authorization.k8s.io/v1 ClusterRoleBinding
W0621 14:39:20.804103 42437 warnings.go:70] admissionregistration.k8s.io/v1beta1 MutatingWebhookConfiguration is deprecated in v1.16+, unavailable in v1.22+; use admissionregistration.k8s.io/v1 MutatingWebhookConfiguration
NAME: chaos
LAST DEPLOYED: Mon Jun 21 14:39:20 2021
NAMESPACE: chaosblade
STATUS: deployed
REVISION: 1
TEST SUITE: None
NOTES:
Thank you for using chaosblade.
[root@s5 k8s]#
# 查看安装结果
[root@s5 k8s]# kubectl get pod -n chaosblade | grep chaosblade
chaosblade-operator-67779995db-cs2lv 1/1 Running 0 4m49s
chaosblade-tool-58ch2 1/1 Running 0 3m27s
chaosblade-tool-qdwb6 1/1 Running 0 2m57s
chaosblade-tool-z8jds 1/1 Running 0 2m57s
[root@s5 k8s]#ChaosBlade-Operator 启动后将会在每个节点部署一个 chaosblade-tool Pod 和一个 chaosblade-operator Pod，如果都运行正常，则安装成功。上面设置 --set webhook.enable=true 是为了 Pod 文件系统 I/O 故障实验，如果不需要进行该实验，则无需添加该设置。
示例应用准备
配置实验对象，这里使用 guestbook 应用，如果你已经有应用了，就不需要安装这个小的示例应用。
示例应用安装
- 添加helm仓库
[root@s5 k8s]# helm repo add apphub-incubator https://apphub.aliyuncs.com/incubator/
"apphub-incubator" has been added to your repositories
- 安装应用示例
[root@s5 k8s]# helm install guestbook apphub-incubator/guestbook --set service.type=NodePort --namespace=chaosblade
NAME: guestbook
LAST DEPLOYED: Mon Jun 21 22:42:41 2021
NAMESPACE: chaosblade
STATUS: deployed
REVISION: 1
TEST SUITE: None
NOTES:
1. Get the application URL by running these commands:
export NODE_PORT=$(kubectl get --namespace chaosblade -o jsonpath="{.spec.ports[0].nodePort}" services guestbook)
export NODE_IP=$(kubectl get nodes --namespace chaosblade -o jsonpath="{.items[0].status.addresses[0].address}")
echo http://$NODE_IP:$NODE_PORT
[root@s5 k8s]# echo http://$NODE_IP:$NODE_PORT
http://172.31.184.225:32310默认的 Service 类型为 LoadBalancer，这里为了方便访问设置为了 NodePort。
示例应用安装验证
访问http://nodeip:nodeport。
成功后可以看到这个界面，输入任何字符点击submit都会显示在上面，如此而已的一个小应用。
模拟Pod 网络丢包场景
目标
对 redis-master-b96c9795b-4ghxq Pod 注入丢包率 50% 的故障，持续10分钟，只针对 IP 为 10.100.53.195 的 pod 生效，也就是除 10.100.53.195 以外的 pod 都能正常访问 redis-master-b96c9795b-4ghxq。
配置
当前网络信息如下：
[root@s5 chaosblade_scenarios]# kubectl get pods -n chaosblade -o wide
NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES
chaosblade-operator-67779995db-ns4qg 1/1 Running 0 100m 10.100.53.193 s7 <none> <none>
chaosblade-tool-bhgbk 1/1 Running 0 100m 172.31.184.224 s7 <none> <none>
chaosblade-tool-mqmrc 1/1 Running 0 100m 172.31.184.226 s6 <none> <none>
chaosblade-tool-xqgl5 1/1 Running 0 100m 172.31.184.225 s5 <none> <none>
guestbook-7fcc447874-q248s 1/1 Running 0 98m 10.100.53.194 s7 <none> <none>
guestbook-7fcc447874-zpbn4 1/1 Running 0 98m 10.100.220.67 s6 <none> <none>
mall-tiny-deployment-85bdb875cf-zl6jw 1/1 Running 0 54m 10.100.220.71 s6 <none> <none>
redis-master-b96c9795b-4ghxq 1/1 Running 0 10m 10.100.53.196 s7 <none> <none>
redis-slave-6b8d456947-c6h64 1/1 Running 0 98m 10.100.53.195 s7 <none> <none>
redis-slave-6b8d456947-twgk9 1/1 Running 0 98m 10.100.220.68 s6 <none> <none>配置文件如下。
[root@s5 chaosblade_scenarios]# cat loss_pod_network_by_names.yaml
apiVersion: chaosblade.io/v1alpha1
kind: ChaosBlade
metadata:
name: loss-pod-network-by-names
spec:
experiments:
- scope: pod
target: network
action: loss
desc: "loss pod network by names"
matchers:
- name: names
value:
- "redis-master-b96c9795b-4ghxq"
- name: namespace
value:
- "chaosblade"
- name: interface
value: ["eth0"]
- name: percent
value: ["50"]
- name: timeout
value: ["600"]
- name: destination-ip
value: ["10.100.53.195"]
[root@s5 chaosblade_scenarios]#执行
[root@s5 chaosblade_scenarios]# kubectl apply -f loss_pod_network_by_names.yaml
chaosblade.chaosblade.io/loss-pod-network-by-names created
[root@s5 chaosblade_scenarios]#验证
1. 登录到10.100.53.195服务器，执行ping命令。
[root@s5 chaosblade_scenarios]# kubectl exec -it redis-slave-6b8d456947-c6h64 bash -n chaosblade
kubectl exec [POD] [COMMAND] is DEPRECATED and will be removed in a future version. Use kubectl exec [POD] -- [COMMAND] instead.
[ root@redis-slave-6b8d456947-c6h64:/data ]$ ping 10.100.53.196
PING 10.100.53.196 (10.100.53.196) 56(84) bytes of data.
64 bytes from 10.100.53.196: icmp_seq=1 ttl=63 time=0.112 ms
64 bytes from 10.100.53.196: icmp_seq=2 ttl=63 time=0.096 ms
64 bytes from 10.100.53.196: icmp_seq=3 ttl=63 time=0.098 ms
64 bytes from 10.100.53.196: icmp_seq=4 ttl=63 time=0.091 ms
64 bytes from 10.100.53.196: icmp_seq=7 ttl=63 time=0.092 ms
64 bytes from 10.100.53.196: icmp_seq=8 ttl=63 time=0.084 ms
64 bytes from 10.100.53.196: icmp_seq=13 ttl=63 time=0.085 ms
64 bytes from 10.100.53.196: icmp_seq=14 ttl=63 time=0.088 ms
64 bytes from 10.100.53.196: icmp_seq=17 ttl=63 time=0.086 ms
^C
--- 10.100.53.196 ping statistics ---
17 packets transmitted, 9 received, 47% packet loss, time 15999ms
rtt min/avg/max/mdev = 0.084/0.092/0.112/0.012 ms
[ root@redis-slave-6b8d456947-c6h64:/data ]$确实产生近50%的丢包。
2. 进入master机器。检查网络队列。
[root@s5 chaosblade_scenarios]# kubectl exec -it redis-master-b96c9795b-4ghxq bash -n chaosblade
kubectl exec [POD] [COMMAND] is DEPRECATED and will be removed in a future version. Use kubectl exec [POD] -- [COMMAND] instead.
root@redis-master-b96c9795b-4ghxq:/data# tc qdisc ls dev eth0
qdisc prio 1: root refcnt 2 bands 4 priomap 1 2 2 2 1 2 0 0 1 1 1 1 1 1 1 1
qdisc netem 40: parent 1:4 limit 1000 loss 50%
root@redis-master-b96c9795b-4ghxq:/data#确实产生队列规则。
可见这个模拟是通过操作qdisc来实现的。如果你有兴趣的话，可以去查一下linux上的traffic control的逻辑。
恢复
[root@s5 chaosblade_scenarios]# kubectl delete -f loss_pod_network_by_names.yaml
chaosblade.chaosblade.io "loss-pod-network-by-names" deleted留个思考的空间：
通过tc命令模拟的丢包和应用问题导致的丢包有什么不同？
本文参与 腾讯云自媒体同步曝光计划，分享自微信公众号。
原始发表：2021-07-02，如有侵权请联系 cloudcommunity@tencent.com 删除
评论
登录后参与评论
推荐阅读
目录
相关产品与服务
容器服务
腾讯云容器服务（Tencent Kubernetes Engine, TKE）基于原生 kubernetes 提供以容器为核心的、高度可扩展的高性能容器管理服务，覆盖 Serverless、边缘计算、分布式云等多种业务部署场景，业内首创单个集群兼容多种计算节点的容器资源管理模式。同时产品作为云原生 Finops 领先布道者，主导开源项目Crane，全面助力客户实现资源优化、成本控制。
腾讯云开发者
