Curator是一个用来管理Elasticsearch索引的工具,使用它可以管理需要删除或保留的索引数据。
使用Curator可以完成以下功能:
为别名(Alias)添加或移除索引
创建索引
删除索引
关闭索引
删除快照
打开已经关闭的索引
更改分片路由配置
强制合并索引
重建索引(包括从远程的集群)
更改索引每个分片的副本数量
为索引创建快照
从快照还原
rollover indices(当某个别名指向的实际索引过大的时候,自动将别名指向下一个实际索引)
详情参考官网https://www.elastic.co/guide/en/elasticsearch/client/curator/current/actions.html
https://www.elastic.co/guide/en/elasticsearch/client/curator/current/version-compatibility.html
2、安装elasticsearch-curator
这里采用pip安装
pip install elasticsearch-curator
curator --version
curator, version 5.8.4
curator_cli --host 10.0.0.1:9200 --http_auth 'user:password' show-indices
curator_cli --host 10.0.0.1:9200 --http_auth 'user:password' delete-indices --filter_list '[{"filtertype": "pattern", "kind": "prefix", "value": "filebeat-"}, {"filtertype": "age", "source": "name", "direction": "older", "timestring": "%Y.%m.%d", "unit": "days", "unit_count": 7}]'
其他的action操作参考https://www.elastic.co/guide/en/elasticsearch/client/curator/current/singleton-cli.html
如您的操作比较复杂,参数太多或不想使用命令行参数,可以将参数放在配置文件中执行。
在指定的 config 目录下,需要编辑 config.yml 和 action.yml 两个配置文件。
config.yml
# Remember, leave a key empty if there is no value. None will be a string,
# not a Python "NoneType"
client:
hosts:
- 10.0.0.1
port: 9200
url_prefix:
use_ssl: False
certificate:
client_cert:
client_key:
ssl_no_validate: False
username: elastic
password: password
timeout: 30
master_only: False
logging:
loglevel: INFO
logfile:
logformat: default
blacklist: ['elasticsearch', 'urllib3']
action.yml举例几个action
delete.action
actions:
1:
action: delete_indices
description: "delete apm indices before 30 days."
options:
ignore_empty_list: True
disable_action: False
continue_if_exception: False
allow_ilm_indices: True
filters:
- filtertype: kibana
exclude: True
- filtertype: pattern
kind: regex
value: '^(\.monitoring-).*$'
- filtertype: age
source: name
direction: older
timestring: '%Y.%m.%d'
unit: days
unit_count: 30
close.action
actions:
1:
action: close
description: "close apm indices before 15 days."
options:
ignore_empty_list: True
disable_action: False
continue_if_exception: False
allow_ilm_indices: True
filters:
- filtertype: kibana
exclude: True
- filtertype: pattern
kind: regex
value: '^(\.monitoring-).*$'
- filtertype: age
source: name
direction: older
timestring: '%Y.%m.%d'
unit: days
unit_count: 15
snapshot.action
actions:
1:
action: snapshot
description: >-
Snapshot indices prefixed indices older than 1 day
options:
repository: backup_s3_repository
name: '<prod-log-backup-{now/d-1d}>'
ignore_unavailable: False
include_global_state: True
partial: False
wait_for_completion: True
skip_repo_fs_check: False
allow_ilm_indices: True
filters:
- filtertype: pattern
kind: regex
value: '^(\.monitoring-).*$'
exclude: True
- filtertype: pattern
kind: regex
value: '.*(stag).*$'
exclude: True
- filtertype: period
source: name
range_from: -1
range_to: -1
timestring: '%Y.%m.%d'
unit: days
delete_snapshot.action
actions:
1:
action: delete_snapshots
description: >-
Delete snapshots from the selected repository older than 365 days
options:
repository: backup_s3_repository
disable_action: False
allow_ilm_indices: True
filters:
- filtertype: pattern
kind: regex
value: '^(prod-log-backup-).*$'
- filtertype: age
source: name
direction: older
timestring: '%Y.%m.%d'
unit: days
unit_count: 365
forcemerge.action
actions:
1:
action: forcemerge
description: >-
forceMerge syslog- prefixed indices older than 2 days (based on index
creation_date) to 2 segments per shard. Delay 120 seconds between each
forceMerge operation to allow the cluster to quiesce. Skip indices that
have already been forcemerged to the minimum number of segments to avoid
reprocessing.
options:
ignore_empty_list: True
max_num_segments: 2
delay: 120
timeout_override:
continue_if_exception: False
filters: - filtertype: pattern
kind: prefix
value: syslog-
exclude: - filtertype: age
source: name
direction: older
timestring: '%Y.%m.%d'
unit: days
unit_count: 2
- filtertype: forcemerged
max_num_segments: 2
actions:
1:
action: create_index
description: "创建索引名为:'nginx-%Y-%m-%d-%H:%M:%S'的索引"
options:
name: nginx-%Y-%m-%d-%H:%M:%S #索引名称
extra_settings: #索引信息
settings:
number_of_shards: 2
number_of_replicas: 1
mappings:
student:
_routing:
required: true
properties:
name:
type: keyword
age:
type: integer
执行命令
curator --config config.yml action.yml
这里以forcemerge 为示例
forcemerge前
forcemerge后
可以清晰看到doc.deleted碎片有所下降
6 0 * * * curator --config /data/elasticsearch-curator/config.yml /data/elasticsearch-curator/action.yml
原创声明:本文系作者授权腾讯云开发者社区发表,未经许可,不得转载。
如有侵权,请联系 cloudcommunity@tencent.com 删除。
原创声明:本文系作者授权腾讯云开发者社区发表,未经许可,不得转载。
如有侵权,请联系 cloudcommunity@tencent.com 删除。