专栏首页二狗的DBA之路ES索引管理工具 - curator
原创

ES索引管理工具 - curator

官方文档:https://www.elastic.co/guide/en/elasticsearch/client/curator/current/actions.html

社区好文:https://blog.csdn.net/laoyang360/article/details/85882832

curator与es版本的兼容性:https://www.elastic.co/guide/en/elasticsearch/client/curator/current/version-compatibility.html

curator允许对索引和快照执行许多不同的操作,包括:

1. 从别名添加或删除索引(或两者!)
2. 更改分片路由分配更改分片路由分配
3. 关闭索引关闭索引
4. 创建索引创建索引
5. 删除索引删除索引
6. 删除快照删除快照
7. 打开被关闭的索引打开被关闭的索引
8. 对索引执行forcemerge段合并操作对索引执行forcemerge段合并操作
9. reindex索引,包括来自远程集群的索引reindex索引,包括来自远程集群的索引
10. 更改索引的每个分片的副本数 更改索引的每个分片的副本数
11. rollover索引rollover索引
12. 生成索引的快照(备份)生成索引的快照(备份)
13. 还原快照还原快照

使用方法

安装

pip install elasticsearch-curator

命令行下单次执行

curator_cli --host 127.0.0.1 --port 9200 close --filter_list '[{"filtertype":"age","source":"creation_date","direction":"older","unit":"days","unit_count":1},{"filtertype":"pattern","kind":"prefix","value":"logs_"}]'

更常用的是,定时任务执行:

mkdir /root/.curator
cd  /root/.curator

cat curator.yml 内容如下:

client:
 hosts: 192.168.2.187
 port: 9200
 url_prefix:
 use_ssl: False
 certificate:
 client_cert:
 client_key:
 ssl_no_validate: False
 http_auth:
 timeout: 30
 master_only: False
logging:
 loglevel: INFO
 logfile: /var/log/curator.log
 logformat: default
 blacklist: ['elasticsearch', 'urllib3']

cat action.yml 内容如下:

# 下面我这里演示的是关闭7天前的索引

actions:
 1:
 action: close
 description: "Close selected indices"
 options:
 ignore_empty_list: True
 disable_action: false
 skip_flush: false
 delete_aliases: false
 filters:
 - filtertype: pattern
 kind: prefix
 value: 'logs_|filebeat-|logstash-services-|metricsbeat-'
 - filtertype: age
 source: name
 direction: older
 timestring: '%Y-%m-%d'
 unit: days
 unit_count: 7

由于支持的action操作非常多,建议直接参考官网配置即可:

https://www.elastic.co/guide/en/elasticsearch/client/curator/current/actions.html
https://www.elastic.co/guide/en/elasticsearch/client/curator/current/examples.html

执行:

格式:curator [--config CONFIG.YML] [--dry-run] ACTION_FILE.YML

curator --config curator.yml action.yml

添加定时任务:

50 23 * * * cd /root/.curator/ && curator --config curator.yml action.yml
tail -F /var/log/curator.log 日志类似如下:
2019-11-04 23:14:13,089 INFO Action ID: 1, "close" completed.
2019-11-04 23:14:13,089 INFO Job completed.
2019-11-04 23:14:30,934 INFO Preparing Action ID: 1, "close"
2019-11-04 23:14:30,935 INFO Creating client object and testing connection
2019-11-04 23:14:30,941 INFO Instantiating client object
2019-11-04 23:14:30,942 INFO Testing client connectivity
2019-11-04 23:14:30,949 INFO Successfully created Elasticsearch client object with provided settings
2019-11-04 23:14:30,953 INFO Trying Action ID: 1, "close": Close selected indices
2019-11-04 23:14:30,995 INFO Closing 1 selected indices: ['metricsbeat-2019-11-01']
2019-11-04 23:14:31,164 INFO Action ID: 1, "close" completed.
2019-11-04 23:14:31,165 INFO Job completed.
2019-11-04 23:18:06,032 INFO Instantiating client object
2019-11-04 23:18:06,033 INFO Testing client connectivity
2019-11-04 23:18:06,041 INFO Successfully created Elasticsearch client object with provided settings

一个组合的写法(也可以将多个action拆分成多个脚本来执行):

actions:
 1:
 action: create_index
 description: "Create index as named"
 options:
  continue_if_exception: True
 name: '<logstash-{now/d+1d}>'
 extra_settings:
 settings:
 number_of_shards: 2
 number_of_replicas: 0
 2:
 action: close
 description: "Close selected indices"
 options:
 ignore_empty_list: True
 disable_action: false
 skip_flush: false
 delete_aliases: false
 filters:
 - filtertype: pattern
 kind: prefix
 value: 'logs_|filebeat-|logstash-services-|metricsbeat-'
 - filtertype: age
 source: name
 direction: older
 timestring: '%Y-%m-%d'
 unit: days
 unit_count: 7
 3:
 action: delete_indices
 description: "Delete selected indices"
 options:
 continue_if_exception: False
 timeout_override: 300
 filters:
 - filtertype: pattern
 kind: prefix
 value: 'logs_|filebeat-|logstash-services-|metricsbeat-'
 - filtertype: age
 source: name
 direction: older
 timestring: '%Y-%m-%d'
 unit: days
 unit_count: 14

脚本正常执行时候,日志如下:

2019-11-05 00:42:06,377 INFO Preparing Action ID: 1, "create_index"
2019-11-05 00:42:06,377 INFO Creating client object and testing connection
2019-11-05 00:42:06,384 INFO Instantiating client object
2019-11-05 00:42:06,385 INFO Testing client connectivity
2019-11-05 00:42:06,392 INFO Successfully created Elasticsearch client object with provided settings
2019-11-05 00:42:06,396 INFO Trying Action ID: 1, "create_index": Create index as named
2019-11-05 00:42:06,396 INFO "<logstash-{now/d+2d}>" is using Elasticsearch date math.
2019-11-05 00:42:06,396 INFO Creating index "<logstash-{now/d+2d}>" with settings: {'settings': {'number_of_shards': 2, 'number_of_replicas': 0}}
2019-11-05 00:42:06,608 INFO Action ID: 1, "create_index" completed.
2019-11-05 00:42:06,608 INFO Preparing Action ID: 2, "close"
2019-11-05 00:42:06,608 INFO Creating client object and testing connection
2019-11-05 00:42:06,609 INFO Instantiating client object
2019-11-05 00:42:06,609 INFO Testing client connectivity
2019-11-05 00:42:06,615 INFO Successfully created Elasticsearch client object with provided settings
2019-11-05 00:42:06,619 INFO Trying Action ID: 2, "close": Close selected indices
2019-11-05 00:42:06,672 INFO Closing 3 selected indices: ['filebeat-2019-10-02', 'filebeat-2018-10-25', 'filebeat-2019-10-25']
2019-11-05 00:42:06,875 INFO Action ID: 2, "close" completed.
2019-11-05 00:42:06,875 INFO Preparing Action ID: 3, "delete_indices"
2019-11-05 00:42:06,875 INFO Creating client object and testing connection
2019-11-05 00:42:06,876 INFO Instantiating client object
2019-11-05 00:42:06,876 INFO Testing client connectivity
2019-11-05 00:42:06,881 INFO Successfully created Elasticsearch client object with provided settings
2019-11-05 00:42:06,885 INFO Trying Action ID: 3, "delete_indices": Delete selected indices
2019-11-05 00:42:06,927 INFO Deleting 2 selected indices: ['filebeat-2018-10-25', 'filebeat-2019-10-02']
2019-11-05 00:42:06,927 INFO ---deleting index filebeat-2018-10-25
2019-11-05 00:42:06,927 INFO ---deleting index filebeat-2019-10-02
2019-11-05 00:42:07,007 INFO Action ID: 3, "delete_indices" completed.
2019-11-05 00:42:07,007 INFO Job completed.

切记:

curator适用于基于时间或者template其他方式创建的索引,不适合单一索引存储N久历史数据的操作的场景。

原创声明,本文系作者授权云+社区发表,未经许可,不得转载。

如有侵权,请联系 yunjia_community@tencent.com 删除。

我来说两句

0 条评论
登录 后参与评论

相关文章

  • LENGTH 和 CHAR_LENGTH 的区别

    char(N) 和 varchar(N) 这里的N指的是字符长度,而不是字节长度。就是说可以插入N个字符的长度的内容(不管插入的是英文还是中文,反正是按照长度...

    二狗不要跑
  • mariadb BSL协议期限

    MariaDB 修改为了BSL协议,不再是GPL协议了。因此我们在生产环境如果要使用的话,需要注意下。

    二狗不要跑
  • nginx配置让任何文件在浏览器中显示文本text/plain

    http://www.sijitao.net/1879.html?utm_source=tuicool&utm_medium=referral

    二狗不要跑
  • 疫情现状分析

    从疫情开始后,全国人民开始了禁足模式,尽量少出门,大家的信息来源都是互联网,通过互联网来了解疫情实时情况。

    用户6825444
  • 复合饼图绘制(Excel绘制图表系列课程)

    分享Excel的图表已经第三周了,曾经说给我好处的人,到现在还没有兑现。 嗯,我怎么会是这么庸俗的人,嗯嗯!!! 今天和大家分享一个相对于前两期都比较容易绘...

    用户1332619
  • Windows 平台下 Linux 的安装

    本文将介绍如何在Windows机器上进行Linux-CentOS的安装,在第一部分我会以截图的形式简述安装的步骤,在第二部分我会就安装时的一些注意事项进行深入讲...

    石璞东
  • 【大结局】《从案例中学习JavaScript》之酷炫音乐播放器(四)

    剽悍一小兔
  • 拨打18687679495_缅甸小勐拉银河国际开户Redis为什么这么快

    Redis是一个开源的内存中的数据结构存储系统,它可以用作:数据库、缓存和消息中间件。

    我是你的眼晶
  • 怎么改变证件照的底色

    证件照除了用于证件外,报名或是投递简历也是需要的,如果此时恰好没有携带电子版照片,附近又没有合适的照相馆该怎么办呢?用手机来拍不就好了,用下边的办法还能换底色。

    互动20
  • kibana日志管理与配置

    老七Linux

扫码关注云+社区

领取腾讯云代金券