Elasticsearch忽略自定义索引模板，并用默认Filebeat索引模板覆盖输出索引的映射。
Elasticsearch忽略自定义索引模板，并用默认Filebeat索引模板覆盖输出索引的映射。
提问于 2022-04-21 22:10:07
你想做什么？
使用File节拍从JSON files in ndjson format获取输入数据为JSON files in ndjson format，并在没有附加键的情况下将它们插入到Elasticsearch的my_index中。
让我看看你的吐露。
Elasticsearch.yml
# ---------------------------------- Cluster -----------------------------------
#
cluster.name: masterCluster
#
# ------------------------------------ Node ------------------------------------
#
node.name: masterNode
#
#----------------------- BEGIN SECURITY AUTO CONFIGURATION -----------------------
# Security features
xpack.security.enabled: false
xpack.security.enrollment.enabled: false
xpack.security.http.ssl.enabled: false
xpack.security.transport.ssl.enabled: false
#----------------------- END SECURITY AUTO CONFIGURATION -------------------------Filebeat.yml
# ============================== Filebeat inputs ===============================
filebeat.inputs:
- type: filestream
enabled: true
paths:
- /home/asura/EBK/data/*.json
parser:
- ndjson:
keys_under_root: true
add_error_key: true
# ======================= Elasticsearch template setting =======================
setup.ilm.enabled: false
setup.template:
name: "my_index_template"
pattern: "my_index*"
# ---------------------------- Elasticsearch Output ----------------------------
output.elasticsearch:
hosts: ["localhost:9200"]
index: "my_index"
my_index和my_index_template是什么样子的？
my_index在Kibana中的映射：
{
"mappings": {}
}在基巴尼的my_index_template预览：
{
"template": {
"settings": {
"index": {
"routing": {
"allocation": {
"include": {
"_tier_preference": "data_content"
}
}
}
}
},
"aliases": {},
"mappings": {}
}
}您的输入文件是什么样子的？
input.json
{"filename" :"16.avi", "frame": 131, "Class":"person", "confidence":32, "Date & Time" :"Thu Oct 3 14:02:41 2019", "Others" :"Blue"}
{"filename" :"16.avi", "frame": 131, "Class":"person", "confidence":36, "Date & Time" :"Thu Oct 3 14:02:41 2019", "Others" :"Grey,Blue"}我把上面的文件拖放到被监视的文件夹中，这样插入就可以了。
插入Elasticsearch之后，数据看起来是什么样子的？
获取请求：http://<host>:<my_port>/my_index/_search?filter_path=hits.hits._source
答复：
{
"hits": {
"hits": [
{
"_source": {
"@timestamp": "2022-04-21T21:49:04.084Z",
"log": {
"offset": 0,
"file": {
"path": "/home/asura/EBK/data/input.json"
}
},
"frame": 131,
"Class": "person",
"input": {
"type": "filestream"
},
"ecs": {
"version": "8.0.0"
},
"host": {
"name": "pisacha"
},
"agent": {
"ephemeral_id": "d389a35d-40f7-4680-a485-8e6939d011ab",
"id": "c6cb1ce5-ff92-499d-9e3c-e79478795fca",
"name": "pisacha",
"type": "filebeat",
"version": "8.1.3"
},
"Date & Time": "Thu Oct 3 14:02:41 2019",
"Others": "Blue",
"filename": "16.avi",
"confidence": 32
}
},
{
"_source": {
"@timestamp": "2022-04-21T21:49:04.084Z",
"agent": {
"type": "filebeat",
"version": "8.1.3",
"ephemeral_id": "d389a35d-40f7-4680-a485-8e6939d011ab",
"id": "c6cb1ce5-ff92-499d-9e3c-e79478795fca",
"name": "pisacha"
},
"Others": "Grey,Blue",
"filename": "16.avi",
"input": {
"type": "filestream"
},
"frame": 131,
"Class": "person",
"ecs": {
"version": "8.0.0"
},
"host": {
"name": "pisacha"
},
"confidence": 36,
"log": {
"offset": 133,
"file": {
"path": "/home/asura/EBK/data/input.json"
}
},
"Date & Time": "Thu Oct 3 14:02:41 2019"
}
},
{
"_source": {
"@timestamp": "2022-04-21T21:49:04.084Z",
"input": {
"type": "filestream"
},
"agent": {
"id": "c6cb1ce5-ff92-499d-9e3c-e79478795fca",
"name": "pisacha",
"type": "filebeat",
"version": "8.1.3",
"ephemeral_id": "d389a35d-40f7-4680-a485-8e6939d011ab"
},
"ecs": {
"version": "8.0.0"
},
"host": {
"name": "pisacha"
},
"message": "",
"error": {
"type": "json",
"message": "Error decoding JSON: EOF"
}
}
}
]
}
}它没有使用我指定的模板。
令人惊讶的是
在my_index插入数据后，在Kibana预览：
{
"mappings": {
"properties": {
"@timestamp": {
"type": "date"
},
"Class": {
"type": "text",
"fields": {
"keyword": {
"type": "keyword",
"ignore_above": 256
}
}
},
"Date & Time": {
"type": "text",
"fields": {
"keyword": {
"type": "keyword",
"ignore_above": 256
}
}
},
"Others": {
"type": "text",
"fields": {
"keyword": {
"type": "keyword",
"ignore_above": 256
}
}
},
"agent": {
"properties": {
"ephemeral_id": {
"type": "text",
"fields": {
"keyword": {
"type": "keyword",
"ignore_above": 256
}
}
},
"id": {
"type": "text",
"fields": {
"keyword": {
"type": "keyword",
"ignore_above": 256
}
}
},
"name": {
"type": "text",
"fields": {
"keyword": {
"type": "keyword",
"ignore_above": 256
}
}
},
"type": {
"type": "text",
"fields": {
"keyword": {
"type": "keyword",
"ignore_above": 256
}
}
},
"version": {
"type": "text",
"fields": {
"keyword": {
"type": "keyword",
"ignore_above": 256
}
}
}
}
},
"confidence": {
"type": "long"
},
"ecs": {
"properties": {
"version": {
"type": "text",
"fields": {
"keyword": {
"type": "keyword",
"ignore_above": 256
}
}
}
}
},
"error": {
"properties": {
"message": {
"type": "text",
"fields": {
"keyword": {
"type": "keyword",
"ignore_above": 256
}
}
},
"type": {
"type": "text",
"fields": {
"keyword": {
"type": "keyword",
"ignore_above": 256
}
}
}
}
},
"filename": {
"type": "text",
"fields": {
"keyword": {
"type": "keyword",
"ignore_above": 256
}
}
},
"frame": {
"type": "long"
},
"host": {
"properties": {
"name": {
"type": "text",
"fields": {
"keyword": {
"type": "keyword",
"ignore_above": 256
}
}
}
}
},
"input": {
"properties": {
"type": {
"type": "text",
"fields": {
"keyword": {
"type": "keyword",
"ignore_above": 256
}
}
}
}
},
"log": {
"properties": {
"file": {
"properties": {
"path": {
"type": "text",
"fields": {
"keyword": {
"type": "keyword",
"ignore_above": 256
}
}
}
}
},
"offset": {
"type": "long"
}
}
},
"message": {
"type": "text",
"fields": {
"keyword": {
"type": "keyword",
"ignore_above": 256
}
}
}
}
}
}my_index_template中的映射是巨大的，有数万行长。就好像它拥有fields.yml所拥有的所有字段一样。此外，默认情况下，它还为它创建了一个名为data_stream的my_index。
即使在设置setup.ilm.enabled: false之后，数据仍然会被插入，所有字段都显示在filebeat默认索引模板中。我已经搜索并尝试了我所能做的一切，我需要一个没有在黑暗中开枪的人给我一些指导。
用于Elasticsearch，Kibana和Filebeat：8.1.3的版本，如果您需要更多信息，请做评论:)
参考资料：
- 解析ndjson：解析器
- 用于使用自定义索引：https://www.elastic.co/guide/en/beats/filebeat/current/elasticsearch-output.html#index-option-es
- 用于使用自定义模板：https://www.elastic.co/guide/en/beats/filebeat/current/configuration-template.html
- 用于过滤响应：https://www.elastic.co/guide/en/elasticsearch/reference/current/common-options.html#common-options-response-filtering
Stack Overflow用户
回答已采纳
发布于 2022-04-22 06:48:00
TLDR；
我不确定是否有一个选项可以阻止Filebeat添加这些字段。
但是您可以在输出中添加一个滤波处理机来删除它们。
# ============================== Filebeat inputs ===============================
filebeat.inputs:
- type: filestream
enabled: true
paths:
- /home/asura/EBK/data/*.json
parser:
- ndjson:
keys_under_root: true
add_error_key: true
# ======================= Elasticsearch template setting =======================
setup.ilm.enabled: false
setup.template:
name: "my_index_template"
pattern: "my_index*"
# ---------------------------- Elasticsearch Output ----------------------------
output.elasticsearch:
hosts: ["localhost:9200"]
index: "my_index"
processors:
- drop_fields:
fields: ["agent", "ecs", "host", ...]如果完全禁用Beats以添加某些字段的选项存在，那么这将是一个更好的选项。我只是不知道。
编辑：
完整的工作解决方案涉及Globally Declared Processors。
filebeat.inputs:
- type: filestream
# Input Processors act during input stage of processing pipeline
processors:
- drop_fields:
fields: ["key1","key2"]
# ---------------------------- Global Processors ------------------
# Global processors for fields that are added later by filebeat
processors:
- drop_fields:
fields: ["agent", "ecs", "input", "log", "host"]参考：
https://discuss.elastic.co/t/filebeat-didnt-drop-some-of-the-fields-like-agent-ecs-etc/243911/2
页面原文内容由Stack Overflow提供。腾讯云小微IT领域专用引擎提供翻译支持
原文链接：
https://stackoverflow.com/questions/71961421
复制相关文章
相似问题
腾讯云开发者
