本文主要研究一下elasticsearch的data-streams
.ds-<data-stream>-<yyyy.MM.dd>-<generation>
示例.ds-web-server-logs-2099.03.07-000034,generation是一个6位的数字,默认从
000001
开始
# Creates a component template for mappings
PUT _component_template/my-mappings
{
"template": {
"mappings": {
"properties": {
"@timestamp": {
"type": "date",
"format": "date_optional_time||epoch_millis"
},
"message": {
"type": "wildcard"
}
}
}
},
"_meta": {
"description": "Mappings for @timestamp and message fields",
"my-custom-meta-field": "More arbitrary metadata"
}
}
# Creates a component template for index settings
PUT _component_template/my-settings
{
"template": {
"settings": {
"index.lifecycle.name": "my-lifecycle-policy"
}
},
"_meta": {
"description": "Settings for ILM",
"my-custom-meta-field": "More arbitrary metadata"
}
}
主要是利用_component_template创建mappings和settings,方面下面创建index_template使用
PUT _index_template/my-index-template
{
"index_patterns": ["my-data-stream*"],
"data_stream": { },
"composed_of": [ "my-mappings", "my-settings" ],
"priority": 500,
"_meta": {
"description": "Template for my time series data",
"my-custom-meta-field": "More arbitrary metadata"
}
}
PUT /_data_stream/my-data-stream-1/
GET /_data_stream/my-data-stream-1
{
"data_streams": [
{
"name": "my-data-stream-1",
"timestamp_field": {
"name": "@timestamp"
},
"indices": [
{
"index_name": ".ds-my-data-stream-1-2023.08.06-000001",
"index_uuid": "ByCb4bPGSEOXfVf3Txpiiw"
}
],
"generation": 1,
"_meta": {
"my-custom-meta-field": "More arbitrary metadata",
"description": "Template for my time series data"
},
"status": "YELLOW",
"template": "my-data-stream",
"ilm_policy": "my-lifecycle-policy",
"hidden": false,
"system": false,
"allow_custom_routing": false,
"replicated": false
}
]
}
POST my-data-stream-1/_doc
{
"@timestamp": "2099-05-06T16:21:15.000Z",
"message": "192.0.2.42 - - [06/May/2099:16:21:15 +0000] \"GET /images/bg.jpg HTTP/1.0\" 200 24736"
}
返回
{
"_index": ".ds-my-data-stream-1-2023.08.06-000001",
"_id": "bHTfyIkBwVE4kI2xm5nL",
"_version": 1,
"result": "created",
"_shards": {
"total": 2,
"successful": 1,
"failed": 0
},
"_seq_no": 0,
"_primary_term": 1
}
POST my-data-stream-1/_search
{ "query": { "match_all": {} } }
filebeat默认output到elasticsearch创建的就是data streams,如果不想使用其自动加载的模版,则可以设置setup.template.enabled=false
,那么创建的则是普通的index。
elasticsearch7.9版本以xpack的形式推出了data streams,主要是针对持续产生的时间序列数据提供了一种更为简单的方式去对索引进行数据切分和统一查询的方式。