The content of this page has been automatically translated by AI. If you encounter any problems while reading, you can view the corresponding content in Chinese.
When deploying services using Tencent Cloud TKE, you can use Filebeat to collect logs from each pod in TKE and write them to a downstream Elasticsearch cluster. Then, you can query and analyze the logs on Kibana. This article explains how to use a Filebeat DaemonSet to collect logs from containers.
Practical Process
The following example demonstrates the process of using Filebeat to collect logs from Nginx pods running in a TKE container cluster with containerd and writing them to Elasticsearch.
Deploying Nginx DaemonSet
First, use an Nginx image from Docker Hub to deploy a DaemonSet named nginx. For more details, please refer to Creating a Simple Nginx Service.
Deploying Filebeat DaemonSet
1. Creating a ConfigMap
1.1 Filebeat uses filebeat.yml as its main configuration file. First, create a ConfigMap with the filebeat.yml configuration.
1.2 Go to Configuration Management > ConfigMap, click New.
1.3 In the newly created ConfigMap, the variable name is filebeat.yml, and the content is the specific configuration details. Below is a simple filebeat.yml configuration.
filebeat.inputs:
- type: log
symlinks: true
paths:
- /var/log/containers/*.log
output.elasticsearch:
hosts: ['x.x.x.x:9200']
username: "elastic"
password: "x.x.x.x"
2. Deploy the Filebeat DaemonSet using a Filebeat image from a public repository.
Search for the required CAM policy as needed, and click to complete policy association.
3. Configure Data Volume
4. Configure Runtime Parameters
5. Configure Mount Point
Note
Description of Data Volumes and Mount Points:
1. Use ConfigMap data volumes to allow the filebeat pod to read the custom-defined filebeat.yml configuration.
2. Use host path /var/log/containers, so that the filebeat pod can read logs from other pods, since other pods' logs will be printed under the host machine path /var/log/containers.
3. Since the pod logs under the host path /var/log/containers are symbolic links pointing to the logs of each pod in the directory /var/log/pods, you need to mount the host path /var/log/containers to the filebeat pod. This is why you need to define symlinks: true in filebeat.yml, because by default, filebeat does not read link files.
6. View logs in Kibana
Enter the Kibana corresponding to the ES cluster defined in filebeat.yml to check if the corresponding index is generated and if the NGINX logs can be viewed normally.
By following the above steps, you can check that the NGINX logs are being collected properly. However, the above configuration collects logs from all pods on the host machine. Sometimes you need to collect logs from only certain pods. How to achieve this?
Deploy a filebeat daemonSet through YML configuration that can obtain Pod metadata
In actual business scenarios, it is usually necessary to use Filebeat to collect logs from multiple pods deployed on the same host. Typically, it's also required to obtain metadata of the collected pods, such as namespace, pod name, tags, etc., to facilitate filtering or searching. To get the pod metadata, the Kubernetes API needs to be called. Filebeat has implemented this feature internally, so you can directly use Filebeat's container input and add_kubernetes_metadata processors to achieve this.
1. In the TKE console, click the YML Create Resource button and use the following YAML configuration to create a Filebeat DaemonSet.
- apiGroups: [""]# "" indicates the core API group
resources:
- namespaces
- pods
- nodes
verbs:
- get
- watch
- list
- apiGroups: ["apps"]
resources:
- replicasets
verbs: ["get", "list", "watch"]
---
apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
name: filebeat
# should be the namespace where filebeat is running
namespace: default
labels:
k8s-app: filebeat
rules:
- apiGroups:
- coordination.k8s.io
resources:
- leases
verbs: ["get", "create", "update"]
---
apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
name: filebeat-kubeadm-config
namespace: default
labels:
k8s-app: filebeat
rules:
- apiGroups: [""]
resources:
- configmaps
resourceNames:
- kubeadm-config
verbs: ["get"]
---
apiVersion: v1
kind: ServiceAccount
metadata:
name: filebeat
namespace: default
labels:
k8s-app: filebeat
---
Note
Configuration instructions:
1. The namespace is default, and it can be directly replaced as needed.
2. Created a service account named filebeat and granted it permissions to access pods list, get pod details, and other interfaces. Filebeat will use this account to get the metadata of the pods.
3. Collect logs from the /var/log/containers/ directory through container input. The container input can collect the stdout and stderr of the containers.
2. View logs in Kibana. You can see that each log contains Kubernetes fields.
3. The above configuration collected logs of all pods on the node where the filebeat pod is located directly through container input. Of course, it also includes the logs of filebeat itself. In real business scenarios, it is usually only necessary to collect the logs of the pods that the business cares about.
Method one: Filter all collected log events through the Definition processor in filebeat.yml, and filter out the log events that are not of interest (for example, filter out logs of certain namespace and pods not of interest through drop event processor);
Method two: Through the Autodiscover Definition in the new filebeat.yml configuration file, collect only the logs of the fixed pods through the Definition template. Below is a simple Autodiscover configuration that only collects the logs of the pod named nginx:
4. By modifying the ConfigMap named filebeat-config and redeploying the pod to make the new configuration take effect, you can see that only the logs of the nginx pod are collected in Kibana.
