在 Grafana Loki 中用 Logging operator 来访问 Kubernetes pod 日志
原创在 Grafana Loki 中用 Logging operator 来访问 Kubernetes pod 日志
原创
philentso
修改于 2022-12-28 17:35:04
修改于 2022-12-28 17:35:04
在 Grafana Loki 中用 Logging operator 来访问 kubernetes pod 日志
通过前一节 Logging operator 基本了解，本节实战部署日志系统来收集容器应用日志！
本示例介绍如何使用 Logging operator 在 Kubernetes 中收集应用程序和容器日志，以及如何将它们发送到 Grafana Loki.
下图概述了系统的工作原理， Logging operator 从应用程序收集日志，选择要转发到输出的日志，并将选定的日志消息发送到输出：
Logging operator
部署 Loki 和 Grafana
- Add the chart repositories of Loki and Grafana using the following commands:
$ helm repo add grafana https://grafana.github.io/helm-charts
"grafana" has been added to your repositories
$ helm repo add loki https://grafana.github.io/loki/charts
"loki" has been added to your repositories
$ helm repo update
Hang tight while we grab the latest from your chart repositories...
...Unable to get an update from the "testchartmuseum" chart repository (http://192.168.86.129:32155):
Get "http://192.168.86.129:32155/index.yaml": dial tcp 192.168.86.129:32155: connect: connection refused
...
...Successfully got an update from the "flagger" chart repository
...Successfully got an update from the "argo" chart repository
Update Complete. ⎈Happy Helming!⎈- 安装 Loki into the logging namespace:
$ helm upgrade --install --create-namespace --namespace logging loki loki/loki
Release "loki" does not exist. Installing it now.
WARNING: This chart is deprecated
NAME: loki
LAST DEPLOYED: Wed Dec 28 03:34:53 2022
NAMESPACE: logging
STATUS: deployed
REVISION: 1
TEST SUITE: None
NOTES:
Verify the application is working by running these commands:
kubectl --namespace logging port-forward service/loki 3100
curl http://127.0.0.1:3100/api/prom/label
// Grafana Loki Documentation:
https://github.com/grafana/loki/tree/master/production/helm- 安装 Grafana into the logging namespace:
$ helm upgrade --install --create-namespace --namespace logging grafana grafana/grafana \
--set "datasources.datasources\\.yaml.apiVersion=1" \
--set "datasources.datasources\\.yaml.datasources[0].name=Loki" \
--set "datasources.datasources\\.yaml.datasources[0].type=loki" \
--set "datasources.datasources\\.yaml.datasources[0].url=http://loki:3100" \
--set "datasources.datasources\\.yaml.datasources[0].access=proxy"
Release "grafana" does not exist. Installing it now.
NAME: grafana
LAST DEPLOYED: Wed Dec 28 03:47:41 2022
NAMESPACE: logging
STATUS: deployed
REVISION: 1
NOTES:
1. Get your 'admin' user password by running:
kubectl get secret --namespace logging grafana -o jsonpath="{.data.admin-password}" | base64 --decode ; echo
2. The Grafana server can be accessed via port 80 on the following DNS name from within your cluster:
grafana.logging.svc.cluster.local
Get the Grafana URL to visit by running these commands in the same shell:
export POD_NAME=$(kubectl get pods --namespace logging -l "app.kubernetes.io/name=grafana,app.kubernetes.io/instance=grafana" -o jsonpath="{.items[0].metadata.name}")
kubectl --namespace logging port-forward $POD_NAME 3000
3. Login with the password from step 1 and the username: admin
#################################################################################
###### WARNING: Persistence is disabled!!! You will lose your data when #####
###### the Grafana pod is terminated. #####
#################################################################################验证 loki 与 grafana 的 Install：
$ kubectl get pods --namespace logging
NAME READY STATUS RESTARTS AGE
grafana-698cbcc54b-v6b8t 1/1 Running 0 77s
loki-0 1/1 Running 0 14m以上说明都 安装 OK!...
安装过程比较详细，安装参数比较简单，不再一一说明！
部署 Logging operator 和 demo application
Install the Logging operator and a demo application to provide sample log messages.
用 Helm 部署 Logging operator
To install the Logging operator using Helm, complete these steps.
Note:
For the Helm-based installation you need Helm v3.2.1 or later.
. Add the chart repository of the Logging operator using the following commands:
$ helm repo add banzaicloud-stable https://kubernetes-charts.banzaicloud.com
"banzaicloud-stable" has been added to your repositories
$ helm repo update
...
...Successfully got an update from the "grafana" chart repository
...Successfully got an update from the "banzaicloud-stable" chart repository
...Successfully got an update from the "bitnami" chart repository
Update Complete. ⎈Happy Helming!⎈. 将 Logging operator 安装到 logging 命名空间中:
$ helm upgrade --install --wait --create-namespace --namespace logging logging-operator banzaicloud-stable/logging-operator
Release "logging-operator" has been upgraded. Happy Helming!
NAME: logging-operator
LAST DEPLOYED: Wed Dec 28 05:55:20 2022
NAMESPACE: logging
STATUS: deployed
REVISION: 2
TEST SUITE: None. 安装演示应用程序及其日志记录定义.
$ helm upgrade --install --wait --create-namespace --namespace logging logging-demo banzaicloud-stable/logging-demo \
--set "loki.enabled=True"
Release "logging-demo" does not exist. Installing it now.
NAME: logging-demo
LAST DEPLOYED: Wed Dec 28 05:59:29 2022
NAMESPACE: logging
STATUS: deployed
REVISION: 1. Validate your deployment.
$ kubectl get pods --namespace logging
NAME READY STATUS RESTARTS AGE
grafana-698cbcc54b-v6b8t 1/1 Running 0 139m
logging-demo-fluentbit-l7pmd 1/1 Running 0 3m46s
logging-demo-fluentd-0 2/2 Running 0 3m46s
logging-demo-fluentd-configcheck-3861c1ba 0/1 Completed 0 7m51s
logging-demo-log-generator-585749975-k6w7s 1/1 Running 0 7m51s
logging-operator-69b48dbf9d-mtbtt 1/1 Running 0 18m
loki-0 . 创建 logging 资源
$ kubectl -n logging apply -f - <<"EOF"
apiVersion: logging.banzaicloud.io/v1beta1
kind: Logging
metadata:
name: default-logging-simple
spec:
fluentd: {}
fluentbit: {}
controlNamespace: logging
EOF
logging.logging.banzaicloud.io/default-logging-simple created
$ kubectl get Logging -n logging
NAME AGE
default-logging-simple 29s
logging-demo 10m注意：只能在 controlNamespace 中使用 ClusterOutput 和 ClusterFlow 资源.
. 创建 Loki output 定义
$ kubectl -n logging apply -f - <<"EOF"
apiVersion: logging.banzaicloud.io/v1beta1
kind: Output
metadata:
name: loki-output
spec:
loki:
url: http://loki:3100
configure_kubernetes_labels: true
buffer:
timekey: 1m
timekey_wait: 30s
timekey_use_utc: true
EOF
output.logging.banzaicloud.io/loki-output created
$ kubectl get Output -n logging
NAME ACTIVE PROBLEMS
logging-demo-output-loki true
loki-output 注意：在生产环境中，请使用较长的 timekey 间隔以避免生成过多对象
. 创建 flow 资源
$ kubectl -n logging apply -f - <<"EOF"
apiVersion: logging.banzaicloud.io/v1beta1
kind: Flow
metadata:
name: loki-flow
spec:
filters:
- tag_normaliser: {}
- parser:
remove_key_name_field: true
reserve_data: true
parse:
type: nginx
match:
- select:
labels:
app.kubernetes.io/name: log-generator
localOutputRefs:
- loki-output
EOF
flow.logging.banzaicloud.io/loki-flow created. 安装 demo 应用程序
$ kubectl -n logging apply -f - <<"EOF"
apiVersion: apps/v1
kind: Deployment
metadata:
name: log-generator
spec:
selector:
matchLabels:
app.kubernetes.io/name: log-generator
replicas: 1
template:
metadata:
labels:
app.kubernetes.io/name: log-generator
spec:
containers:
- name: nginx
image: banzaicloud/log-generator:0.3.2
EOF
deployment.apps/log-generator created. Validate your deployment
$ kubectl get po -n logging
NAME READY STATUS RESTARTS AGE
default-logging-simple-fluentbit-cglgc 1/1 Running 0 7m52s
default-logging-simple-fluentd-0 2/2 Running 0 7m52s
default-logging-simple-fluentd-configcheck-8677d9e7 0/1 Completed 0 119s
grafana-698cbcc54b-v6b8t 1/1 Running 0 149m
log-generator-cf689c55d-8xrdp 1/1 Running 0 46s
logging-demo-fluentbit-l7pmd 1/1 Running 0 13m
logging-demo-fluentd-0 2/2 Running 0 13m
logging-demo-fluentd-configcheck-d8327387 0/1 Completed 0 119s
logging-demo-log-generator-585749975-k6w7s 1/1 Running 0 17m
logging-operator-69b48dbf9d-mtbtt 1/1 Running 0 28m
loki-0 Validate the deployment
Grafana Dashboard
- 使用下面的命令来检索 Grafana admin 用户的密码:
$ kubectl get secret --namespace logging grafana -o jsonpath="{.data.admin-password}" | base64 --decode ; echo
a6Uo92.........2tnh- 启用到 Grafana 服务的端口转发.
// 也可设置 svc 为 NodePort 访问
$ kubectl -n logging port-forward svc/grafana 3000:80- Open the Grafana Dashboard: http://localhost:3000
- 使用步骤1中检索到的 admin 用户名和密码来登录.
- 选择菜单 > Explore, 选择 Data source > Loki, 然后选择 Log labels > namespace > logging， 应该出现一个日志列表.
类似于以下界面：
log_dis
注意：
如果页面没有得到预期的日志结果，可以在 故障排除 部分找到帮助...
原创声明：本文系作者授权腾讯云开发者社区发表，未经许可，不得转载。
如有侵权，请联系 cloudcommunity@tencent.com 删除。
原创声明：本文系作者授权腾讯云开发者社区发表，未经许可，不得转载。
如有侵权，请联系 cloudcommunity@tencent.com 删除。
评论
登录后参与评论
推荐阅读