专栏首页运维Elasticsearch集群数据迁移

Elasticsearch集群数据迁移

参考

https://www.elastic.co/guide/en/elasticsearch/reference/5.0/modules-snapshots.html https://www.elastic.co/guide/en/elasticsearch/guide/current/_rolling_restarts.html https://blog.csdn.net/u014431852/article/details/52905821

环境

阿里云elasticsearch集群5.0版本 微软云elasticsearch集群5.6版本

需求

需要把阿里云elasticsearch集群新老数据迁移到微软云elasticsearch集群

解决

新数据比较好弄数据源输出到新的微软云kafka集群然后微软云logstash消费新数据到新elasticsearch集群即可,关于老数据迁移比较麻烦,但官网也给了成熟的解决方案既是快照备份与还原,下面实施过程既是对实施过程的记录

实施

阿里云elasticsearch集群操作

一,先关闭数据平衡,注意一个一个的来,关一个节点的进程none,all循环一次,否则最后集群切片变动,恢复时间很长

1、修改elasticsearch.yml配置,添加如下 path.repo: /storage/esdata 设置索引备份快照路径 注意所有的master节点与data节点都需要配置

2、关闭自动平衡

curl -XPUT http://10.10.88.86:9200/_cluster/settings -d' { "transient" : { "cluster.routing.allocation.enable" : "none" } }' elasticsearch日志 [root@elk-es01 storage]# tail -f es-cluster.log [2018-05-11T15:24:15,605][INFO ][o.e.c.s.ClusterSettings ] [elk-es01] updating [cluster.routing.allocation.enable] from [ALL] to [none]

3、重启elasticseach

4、启动自动平衡

curl -XPUT http:///10.10.88.86:9200/_cluster/settings -d' { "transient" : { "cluster.routing.allocation.enable" : "all" } }'

5、重复循环1、2、3、4操作在剩下几个节点,最好通过/_cat/health查看集群恢复100%后操作下一次,如果不关闭allocation重启elasticsearch集群恢复时间很长会从0%开始,关闭后从节点数减一除以总节点数比例开始恢复会块很多

二、设置快照基本配置

curl -XPUT http:///10.10.88.86:9200/_snapshot/client_statistics -d' { "type": "fs", "settings": { "location": "/storage/esdata", "compress": true, "max_snapshot_bytes_per_sec" : "50mb", "max_restore_bytes_per_sec" : "50mb" } }'

注意错误 报错没有权限

store location [/storage/esdata] is not accessible on the node [{elk-es04} {"error":{"root_cause":[{"type":"repository_verification_exception","reason":"[client_statistics] [[TgGhv7V1QGagb_PNDyXM-w, 'RemoteTransportException[[elk-es04][/10.10.88.89:9300][internal:admin/repository/verify]]; nested: RepositoryVerificationException[[client_statistics] store location [/storage/esdata] is not accessible on the node [{elk-es04}{TgGhv7V1QGagb_PNDyXM-w}{7dxKWcF3QreKMZOKhTFgeg}{/10.10.88.89}{/10.10.88.89:9300}]]; nested: AccessDeniedException[/storage/esdata/tests-_u6h8XrJQ32dYPaC7zFC_w/data-TgGhv7V1QGagb_PNDyXM-w.dat];']]"}],"type":"repository_verification_exception","reason":"[client_statistics] [[TgGhv7V1QGagb_PNDyXM-w, 'RemoteTransportException[[elk-es04][10.51.57.54:9300][internal:admin/repository/verify]]; nested: RepositoryVerificationException[[client_statistics] store location [/storage/esdata] is not accessible on the node [{elk-es04}{TgGhv7V1QGagb_PNDyXM-w}{7dxKWcF3QreKMZOKhTFgeg}{/10.10.88.89}{/10.10.88.89:9300}]]; nested: AccessDeniedException[/storage/esdata/tests-_u6h8XrJQ32dYPaC7zFC_w/data-TgGhv7V1QGagb_PNDyXM-w.dat];']]"},"status":500}

解决方法: 发现node1,node2,node3的es权限是500,node4的权限是501,最近比较背,感觉任何一个小问题都会遇到很多意想不到的地方,这不google了下https://discuss.elastic.co/t/why-does-creating-a-repository-fail/22697/16显然是权限问题,nfs这边有些策略,权衡利弊想短时间内解决问题 打算修改node4的es的uid和gid都为500,使其与前三个节点保持一致 https://blog.csdn.net/train006l/article/details/79007483 usermod -u 500 es groupmod -g 500 es 注意usermod时需要es用户没有进程在活动,就是说elasticsearch进程需要关闭,重复上面的node,all操作,另外如果有其他用户占用了500 id,相应的修改且注意相关进程。 如果不放心也可手动修改,主要是/home/es下的权限及es数据权限 find / -user 501 -exec chown -h es {} \; find / -group 501 -exec chown -h es {} \;

再次执行就不报错了: [root@elk-es01 ~]# curl -XPUT http:///10.10.88.86:9200/_snapshot/client_statistics -d'

{ "type": "fs", "settings": { "location": "/storage/esdata", "compress" : "true", "max_snapshot_bytes_per_sec" : "50mb", "max_restore_bytes_per_sec" : "50mb" } }' {"acknowledged":true}[root@elk-es01 ~]#

查看配置

[root@elk-es01 ~]# curl -XGET http:///10.10.88.86:9200/_snapshot/client_statistics?pretty { "client_statistics" : { "type" : "fs", "settings" : { "location" : "/storage/esdata", "max_restore_bytes_per_sec" : "50mb", "compress" : "true", "max_snapshot_bytes_per_sec" : "50mb" } } }

三、给需要迁移的索引做快照

注意索引数量多但是数据量不大时可以统配多一些index,保证每次迁移的数据量不至于太大,比如每次100G以内,防止网络等其他原因导致传输中断等 [root@elk-es01 ~]# curl -XPUT http://10.10.88.86:9200/_snapshot/client_statistics/logstash-nginx-accesslog-2018.05.11 -d'

{ "indices": "logstash-nginx-accesslog-2018.05.11" }' {"accepted":true}[root@elk-es01 ~]#

2018年5月份只是前10天

curl -XPUT http://10.10.88.86:9200/_snapshot/client_statistics/logstash-nginx-accesslog-2018.05 -d' { "indices": "logstash-nginx-accesslog-2018.05.0" }' curl -XPUT http://10.10.88.86:9200/_snapshot/client_statistics/logstash-nginx-accesslog-2018.04 -d' { "indices": "logstash-nginx-accesslog-2018.04" }' curl -XPUT http://10.10.88.86:9200/_snapshot/client_statistics/logstash-nginx-accesslog-2018.03 -d' { "indices": "logstash-nginx-accesslog-2018.03" }'curl -XPUT http://10.10.88.86:9200/_snapshot/client_statistics/logstash-nginx-accesslog-2018.02 -d' { "indices": "logstash-nginx-accesslog-2018.02" }' curl -XPUT http://10.10.88.86:9200/_snapshot/client_statistics/logstash-nginx-accesslog-2018.01 -d' { "indices": "logstash-nginx-accesslog-2018.01" }' curl -XPUT http://10.10.88.86:9200/_snapshot/client_statistics/logstash-nginx-accesslog-2017.12 -d' { "indices": "logstash-nginx-accesslog-2017.12" }' curl -XPUT http://10.10.88.86:9200/_snapshot/client_statistics/logstash-nginx-accesslog-2017.11 -d' { "indices": "logstash-nginx-accesslog-2017.11*" }'

例子

[root@elk-es01 ~]# curl -XPUT http://10.10.88.86:9200/_snapshot/client_statistics/logstash-nginx-accesslog-2017.11 -d' { "indices": "logstash-nginx-accesslog-2017.11*" }' {"accepted":true}[root@elk-es01 ~]# [root@elk-es01 ~]# curl -XGET http://10.10.88.86:9200/_snapshot/client_statistics/logstash-nginx-accesslog-2017.11?pretty { "snapshots" : [ { "snapshot" : "logstash-nginx-accesslog-2017.11", "uuid" : "PwPlyCbQQliZY3saog45LA", "version_id" : 5000299, "version" : "5.0.2", "indices" : [ "logstash-nginx-accesslog-2017.11.20", "logstash-nginx-accesslog-2017.11.17", "logstash-nginx-accesslog-2017.11.24", "logstash-nginx-accesslog-2017.11.30", "logstash-nginx-accesslog-2017.11.22", "logstash-nginx-accesslog-2017.11.18", "logstash-nginx-accesslog-2017.11.15", "logstash-nginx-accesslog-2017.11.16", "logstash-nginx-accesslog-2017.11.27", "logstash-nginx-accesslog-2017.11.26", "logstash-nginx-accesslog-2017.11.19", "logstash-nginx-accesslog-2017.11.21", "logstash-nginx-accesslog-2017.11.28", "logstash-nginx-accesslog-2017.11.23", "logstash-nginx-accesslog-2017.11.25", "logstash-nginx-accesslog-2017.11.29" ], "state" : "IN_PROGRESS", "start_time" : "2018-05-14T02:31:58.900Z", "start_time_in_millis" : 1526265118900, "failures" : [ ], "shards" : { "total" : 0, "failed" : 0, "successful" : 0 } } ] }

注意state 在IN_PROGERESS,变成SUCCESS时快照完成,注意SUCCESS时再执行下一次快照,如果index比较少时也可以一次性执行,不分开。

在微软云elasticsearch集群上操作

四、迁移数据到微软云elasticsearch集群

1、挂载nfs服务端

yum -y install nfs-utils mkdir -p /storage/esdata mount -t nfs 192.168.88.20:/data/es-data01/backup /storage/esdata/ df -h 所有master节点与node节点都需要挂载nfs scp阿里云的数据包到微软云nfs服务端,注意需要先tar包,后解压缩,scp前需要open***打通 scp test@10.10.88.89:/storage/esdata/es20180514.tar.gz ./ scp test@10.10.88.89:/storage/esdata/indices20180514.tar.gz ./

2、修改新集群配置

1)、修改elasticsearch.yml配置,添加如下 path.repo: /storage/esdata #设置索引备份快照路径 注意所有的master节点与data节点都需要配置 2)、关闭自动平衡操作同上

curl -XPUT http://192.168.88.24:9200/_cluster/settings -d' { "transient" : { "cluster.routing.allocation.enable" : "none" } }' sleep 3 /etc/init.d/elasticsearch restart curl -XPUT http://192.168.88.24:9200/_cluster/settings -d' { "transient" : { "cluster.routing.allocation.enable" : "all" } }' curl http://192.168.88.20:9200/_cluster/health?pretty

3、恢复快照

1)、先建数据仓库

curl -XPUT http://192.168.88.20:9200/_snapshot/client_statistics -d' { "type": "fs", "settings": { "location": "/storage/esdata", "compress" : "true", "max_snapshot_bytes_per_sec" : "50mb", "max_restore_bytes_per_sec" : "50mb" } }'

2)、再恢复index

root@prod-elasticsearch-master-01 es]# curl -XPOST http://192.168.88.20:9200/_snapshot/client_statistics/logstash-nginx-accesslog-2018.05/_restore {"accepted":true}

依次执行:注意只有当前任务执行完了,才能正确执行下一个任务,开始index状态是yellow,等都变成green就正常了

curl -XPOST http://192.168.88.20:9200/_snapshot/client_statistics/logstash-nginx-accesslog-2018.04/_restore curl -XPOST http://192.168.88.20:9200/_snapshot/client_statistics/logstash-nginx-accesslog-2018.03/_restore curl -XPOST http://192.168.88.20:9200/_snapshot/client_statistics/logstash-nginx-accesslog-2018.02/_restore curl -XPOST http://192.168.88.20:9200/_snapshot/client_statistics/logstash-nginx-accesslog-2018.01/_restore curl -XPOST http://192.168.88.20:9200/_snapshot/client_statistics/logstash-nginx-accesslog-2017.12/_restore curl -XPOST http://192.168.88.20:9200/_snapshot/client_statistics/logstash-nginx-accesslog-2017.11/_restore

3)、最后检查状态

[root@prod-elasticsearch-master-01 es]# curl -XGET http://192.168.88.20:9200/_snapshot/client_statistics/logstash-nginx-accesslog-2018.05/_status?pretty { "snapshots" : [ { "snapshot" : "logstash-nginx-accesslog-2018.05", "repository" : "client_statistics", "uuid" : "VL9LHHUKTNCHx-xsVJD_eA", "state" : "SUCCESS", "shards_stats" : { "initializing" : 0, "started" : 0, "finalizing" : 0, "done" : 45, "failed" : 0, "total" : 45 }, "stats" : { "number_of_files" : 4278, "processed_files" : 4278, "total_size_in_bytes" : 22892376668, "processed_size_in_bytes" : 22892376668, "start_time_in_millis" : 1526280514416, "time_in_millis" : 505655 }, "indices" : { "logstash-nginx-accesslog-2018.05.08" : { "shards_stats" : { "initializing" : 0, "started" : 0, "finalizing" : 0, "done" : 5, "failed" : 0, "total" : 5 }, "stats" : { "number_of_files" : 524, "processed_files" : 524, "total_size_in_bytes" : 2617117488, "processed_size_in_bytes" : 2617117488, "start_time_in_millis" : 1526280514420, "time_in_millis" : 260276 }, "shards" : { "0" : { "stage" : "DONE", "stats" : { "number_of_files" : 67, "processed_files" : 67, "total_size_in_bytes" : 569057817, "processed_size_in_bytes" : 569057817, "start_time_in_millis" : 1526280514420, "time_in_millis" : 68086 } }, "1" : { "stage" : "DONE", "stats" : { "number_of_files" : 124, "processed_files" : 124, "total_size_in_bytes" : 499182013, "processed_size_in_bytes" : 499182013, "start_time_in_millis" : 1526280514446, "time_in_millis" : 62925 } }, "2" : { "stage" : "DONE", "stats" : { "number_of_files" : 109, "processed_files" : 109, "total_size_in_bytes" : 478469125, "processed_size_in_bytes" : 478469125, "start_time_in_millis" : 1526280698072, "time_in_millis" : 76624 } }, "3" : { "stage" : "DONE", "stats" : { "number_of_files" : 124, "processed_files" : 124, "total_size_in_bytes" : 546347244, "processed_size_in_bytes" : 546347244, "start_time_in_millis" : 1526280653094, "time_in_millis" : 103590 } }, "4" : { "stage" : "DONE", "stats" : { "number_of_files" : 100, "processed_files" : 100, "total_size_in_bytes" : 524061289, "processed_size_in_bytes" : 524061289, "start_time_in_millis" : 1526280514456, "time_in_millis" : 69113 } } } }, "logstash-nginx-accesslog-2018.05.09" : { "shards_stats" : { "initializing" : 0, "started" : 0, "finalizing" : 0, "done" : 5, "failed" : 0, "total" : 5 }, "stats" : { "number_of_files" : 425, "processed_files" : 425, "total_size_in_bytes" : 2436583034, "processed_size_in_bytes" : 2436583034, "start_time_in_millis" : 1526280514425, "time_in_millis" : 505646 }, "shards" : { "0" : { "stage" : "DONE", "stats" : { "number_of_files" : 94, "processed_files" : 94, "total_size_in_bytes" : 462380313, "processed_size_in_bytes" : 462380313, "start_time_in_millis" : 1526280971948, "time_in_millis" : 48123 } }, "1" : { "stage" : "DONE", "stats" : { "number_of_files" : 103, "processed_files" : 103, "total_size_in_bytes" : 506505727, "processed_size_in_bytes" : 506505727, "start_time_in_millis" : 1526280851761, "time_in_millis" : 69562 } }, "2" : { "stage" : "DONE", "stats" : { "number_of_files" : 73, "processed_files" : 73, "total_size_in_bytes" : 506830214, "processed_size_in_bytes" : 506830214, "start_time_in_millis" : 1526280514425, "time_in_millis" : 60508 } }, "3" : { "stage" : "DONE", "stats" : { "number_of_files" : 52, "processed_files" : 52, "total_size_in_bytes" : 494390868, "processed_size_in_bytes" : 494390868, "start_time_in_millis" : 1526280593311, "time_in_millis" : 52673 } }, "4" : { "stage" : "DONE", "stats" : { "number_of_files" : 103, "processed_files" : 103, "total_size_in_bytes" : 466475912, "processed_size_in_bytes" : 466475912, "start_time_in_millis" : 1526280583835, "time_in_millis" : 64169 } } } } } } ] }

本文参与腾讯云自媒体分享计划,欢迎正在阅读的你也加入,一起分享。

我来说两句

0 条评论
登录 后参与评论

相关文章

  • vSphere初体验之新建虚拟机

    物理机:DELL  IPTIPLEX 990 PC机 I52400处理器支持VT功能  8G内存 千兆网卡

    三杯水Plus
  • 代理服务器Tengine的研究与测试

    1)Nginx(发音同 engine x)是一款轻量级的Web 服务器/反向代理服务器及电子邮件(IMAP/POP3)代理服务器,并在一个BSD-like 协议...

    三杯水Plus
  • k8s集群之kubernetes-dashboard和kube-dns组件部署安装

    k8s集群之kubernetes-dashboard和kube-dns组件部署安装

    三杯水Plus
  • SDN开发笔记(三):ODL源码编译生成发行版控制器

    前言 鉴于网上对于SDN开发相关的资料较少又乱的现状,从这篇文章开始,我将陆续分享我在SDN开发过程中的经验,我的SDN项目开发是基于OpenDaylight...

    SDNLAB
  • Hive-基础

    DataScience
  • 高质量代码-智慧城市GIS平台后端代码

    《高质量代码-智慧城市GIS平台数据表设计》一文介绍了项目中的数据库表设计。此文介绍优良合理的表设计给后端接口开发带来的便利性。

    MiaoGIS
  • 使用supervisor对程序的输出重定向

    超级大猪
  • 在SAP云平台上部署和运行Docker应用

    容器技术,Docker,虚拟化,这些名词诞生尽管有很长一段时间了,但是在云原生开发领域仍旧热度不减。甚至连SAP赖以成名的ABAP Netweaver,如今也踏...

    Jerry Wang
  • 整个国庆假期总算整理好了全网最新最简单的 OpenJDK13 代码编译

    最近因写文章需要查看 JVM 源码,有时代码逻辑不是很清晰,找半天,趁国庆假期,抽空下载了 OpenJDK13 搭建了 JVM debug 环境,把操作记录写在...

    java架构师
  • apt 安装 nginx

    我这里的ubuntu的环境是18.10。通过上面的表格可以了解到我这里是支持安装的。

    魔王卷子

扫码关注云+社区

领取腾讯云代金券