ElastiFlow提供网络流量数据收集和可视化 | 包括k8s的pod流量分析哦
ElastiFlow提供网络流量数据收集和可视化 | 包括k8s的pod流量分析哦
应用场景
分析链路下的实时主机的流量交互
部署
准备
安装 Java
yum -y install java-openjdk-devel java-openjdk安装 ELK，这里选择7.3.2版本，下载安装包上传至服务器中
elasticsearch-7.3.2-x86_64.rpm
kibana-7.3.2-x86_64.rpm
logstash-7.3.2.rpm使用 rpm 安装
rpm -ivh elasticsearch-7.3.2-x86_64.rpm logstash-7.3.2.rpm kibana-7.3.2-x86_64.rpm重新加载系统服务并开机自启动
systemctl daemon-reload
systemctl enable elasticsearch.service
systemctl enable kibana.service
systemctl enable logstash.service修改 ES 配置文件vim /etc/elasticsearch/elasticsearch.yml
查看 JVM 内存配置vim /etc/elasticsearch/jvm.options最小内存与最大内存使用量，根据系统配置进行调整
修改kibana配置
vim /etc/kibana/kibana.yml
重启服务
systemctl restart elasticsearch.service
systemctl restart kibana.serviceWeb 端访问 http://x.x.x.x:5601 kibana
安装 logstash 模块
/usr/share/logstash/bin/logstash-plugin install logstash-codec-sflow
/usr/share/logstash/bin/logstash-plugin install logstash-codec-netflow /usr/share/logstash/bin/logstash-plugin install logstash-input-udp /usr/share/logstash/bin/logstash-plugin install logstash-input-tcp /usr/share/logstash/bin/logstash-plugin install logstash-filter-dns /usr/share/logstash/bin/logstash-plugin install logstash-filter-geoip /usr/share/logstash/bin/logstash-plugin install logstash-filter-translate下载第三方 Elastiflow
cd /tmp/
git clone https://github.com/robcowart/elastiflow.gitcd /tmp/
wget https://github.com/robcowart/elastiflow/archive/master.zip
unzip master.zip修改 Logstash 内存使用率并复制 Elastiflow 文件夹至相关目录
cp -a elastiflow-master/logstash/elastiflow/. /etc/logstash/elastiflow/
cp -a elastiflow-master/logstash.service.d/. /etc/systemd/system/logstash.service.d/配置 Logstash pipline 文件vim /etc/logstash/pipelines.yml
# This file is where you define your pipelines. You can define multiple.
# For more information on multiple pipelines, see the documentation:
# https://www.elastic.co/guide/en/logstash/current/multiple-pipelines.html
- pipeline.id: main
path.config: "/etc/logstash/conf.d/*.conf"
- pipeline.id: elastiflow
path.config: "/etc/logstash/elastiflow/conf.d/*.conf"执行系统脚本
/usr/share/logstash/bin/system-install开启 Logstash 服务
systemctl daemon-reload
systemctl enable logstash
systemctl start logstash如虚机上启动异常，需要注意 Swap 空间问题，查看 Logstash 模块启动日志
tail -f /var/log/logstash/logstash-plain.log进入下载的文件中找到对应的 Kibana 模版文件下载到本地/tmp/elastiflow-master/kibana/
导入 Kibana
上传后的模版
配置 Kibana
- Pin filters by default: true
- Store URLs in session storage: true
- Time picker quick ranges: （替换成 Git 链接里的配置）https://github.com/robcowart/elastiflow
各协议配置端口可以在 elastiflow.conf 文件找到 cat /etc/systemd/system/logstash.service.d/elastiflow.conf
Cisco 交换机 Sflow 配置
feature sflow
sflow collector-ip 10.16.8.38 vrf management
sflow agent-ip 10.23.224.10
sflow data-source interface Ethernet1/35
sflow data-source interface Ethernet1/36
sflow collector-port 6343查看流量
K8S 容器内 Pod 流量分析
参考链接：https://sflow.net/documentation.php
下载对应 linux 系统的 rpm 安装包
安装
yum install -y hsflowd-centos7-2.0.25-3.x86_64.rpm配置 Sflow Server Controller IP
vim /etc/hsflowd.confsflow {
agent.CIDR = 10.0.0.0/8
polling = 10
sampling = 1000
collector { ip = 172.32.4.65 UDPPort=6343 }
pcap { dev = docker0 }
docker {}
}设置开机启动并启动服务
systemctl enable hsflowd
systemctl start hsflowd查看服务状态
systemctl status hsflowd测试在 192.168.70.38 机器上使用 nping 模拟 tcp 访问对应 Node 节点上的其中一个 Pod 10.17.93.4
ElastiFlow 中查看和 192.168.70.38 有交互的 IP，可以看到 10.17.93.4 Pod IP 地址。