Elasticsearch基于nfs备份环境搭建
原创Elasticsearch基于nfs备份环境搭建
原创
雨夜v1
修改于 2021-03-12 10:25:39
修改于 2021-03-12 10:25:39
前言
从事DBA工作多年，一直没有给大家留下东西。昨天看了铭毅天下老师的文章，感觉我也要写点东西。所以立志每天写一篇文章，在社区留下点痕迹。本文主要分两部分，一部分为nfs环境的搭建，下次会介绍下hdfs环境的搭建；第二部分是关于索引和全库备份方案，详细请查看下文。
1.基础环境介绍
Linux版本 | Elasticsearch版本 |
|---|---|
Redhat 7.6 | Elasticsearch 7.2 |
2.搭建步骤
ip 列表
序号 | ip列表 |
|---|---|
1 | 192.168.248.231 |
2 | 192.168.248.232 |
3 | 192.168.248.233 |
2.1安装依赖包（所有机器）
yum install nfs-utils rpcbind
service nfs start
service rpcbind start
systemctl enable rpcbind; systemctl enable nfs2.2 改配置（192.168.248.231）
vi /etc/exports
/backup/zxb_backup 192.168.*.* (rw,sync,fsid=0)2.3 挂共享卷（192.168.248.232、192.168.248.233）
vim /etc/fstab
192.168.248.231:/backup/zxb_backup /backup/zxb_backup nfs defaults 0 0
mount -a2.4 增加ES配置并重启整套集群
vim /usr/local/elasticsearch/config/elasticsearch.yml
path.repo: ["/backup/zxb_backup/"]3.ES基于索引和全库的备份恢复方案
3.1 基于索引和全库的备份
3.1.1 注册仓储（shakespeare_20200102为仓储名）
curl -uelastic:"xxx" -H "Content-Type: application/json" -XPUT http://192.168.248.231:9201/_snapshot/shakespeare_20200102 -d '{"type":"fs","settings":{"location":"'/backup/zxb_backup/shakespeare_20200102'","compress":true}}'3.1.2 备份shakespeare索引
curl -XPUT -uelastic:"xxx" http://192.168.248.231:9201/_snapshot/shakespeare_20200102/shakespeare?wait_for_completion=true -H "Content-Type:application/json" -d '{"indices": "'shakespeare'","ignore_unavailable": true,"include_global_state": false}'3.1.3 自动化备份全库脚本(backup_es_all_database.sh)
#!/bin/bash
#ES备份全库脚本
#每天定时备份一次，最多保留3天的数据备份
#Author:yuye-v1
#Date:2020.04.21
#vesion 1.0
#加载pofile
if [ -e /etc/pofile ] && [ -e /home/oper/.bash_profile ];then
. /etc/pofile && . /home/oper/.bash_profile
fi
basepath=$(cd "$(diname "$0")";pwd)
INS=`cat ${basepath}/database.txt|grep -v '^#'|sed 's#^$##g'`
db_name=`echo $INS |awk -F: '{print $1}'`
dbip=`echo $INS |awk -F: '{print $2}'`
dbport=`echo $INS |awk -F: '{print $3}'`
dbuser=`echo $INS |awk -F: '{print $4}'`
dbpass=`echo $INS |awk -F: '{print $5}'|sed s#'"'##g`
dbvesion=`echo $INS |awk -F: '{print $6}'|sed s#'"'##g`
if [[ ${dbvesion} = '5.6.15' || ${dbversion} = '6.3.0' ]];then
ESG="curl -k -u ${dbuser}:${dbpass} -XGET https://${dbip}:${dbport}"
ESP="curl -k -u ${dbuser}:${dbpass} -XPUT https://${dbip}:${dbport}"
ESO="curl -k -u ${dbuser}:${dbpass} -XPOST https://${dbip}:${dbport}"
ESD="curl -k -u ${dbuser}:${dbpass} -XDELETE https://${dbip}:${dbport}"
else
ESG="curl -u ${dbuser}:${dbpass} -XGET http://${dbip}:${dbport}"
ESP="curl -u ${dbuser}:${dbpass} -XPUT http://${dbip}:${dbport}"
ESO="curl -u ${dbuser}:${dbpass} -XPOST http://${dbip}:${dbport}"
ESD="curl -u ${dbuser}:${dbpass} -XDELETE http://${dbip}:${dbport}"
fi
LOG_DIR=/applog/es_backup
test -d ${LOG_DIR} || mkdir -p ${LOG_DIR}
LOG_FILE=${LOG_DIR}/backup-$(date +%Y%m%d).log
#检查ES状态
db_status=`${ESG}/_cluster/health?pretty|grep status|tail -n 1|awk -F: '{print $2}'|sed 's/"//g'|sed 's/,//g'|sed 's#[[:space:]]##g'`
if [[ ${db_status} == "green" ]] || [[ ${db_status} == "yellow" ]];then
echo "DB is health!" >>${LOG_FILE}
else
echo "DB is not health!" >>${LOG_FILE}
exit 1
fi
LOG_DIR=/applog/es_backup
test -d ${LOG_DIR} || mkdi -p ${LOG_DIR}
BACKUP_DIR=/backup/zxb_backup
LOG_FILE=${LOG_DIR}/backup-$(date +%Y%m%d).log
echo ">>>Stat shell : ${CURRENT_TIME}" >> ${LOG_FILE}
#step1
#备份当天索引
echo ">>>>Begin backup index " >> ${LOG_FILE}
INDICES=`${ESG}/_cat/indices/*?h=index&s=index:asc -H "Content-Type:application/json"`
#备份文件夹按时间命名（精确到秒）
BACKUP_TIME=$(date +%Y%m%d%H%M%S)
BACKUP_LOCATION=${BACKUP_DIR}/${db_name}-${BACKUP_TIME}
for index in ${INDICES}
do
echo ">>>>Begin to backup ${index}" >> ${LOG_FILE}
if [ ! ${index} ] || [ "${index}" = "" ];then
echo ">>>>${index} is null " >> ${LOG_FILE}
else
RESP_RESULT=`${ESP}/_snapshot/${db_name}-${BACKUP_TIME} -H "Content-Type:application/json" -d '{"type":"fs","settings":{"location":"'${BACKUP_LOCATION}'","compess":true}}'`
echo "${RESP_RESULT}" >> ${LOG_FILE}
SNAP_RESULT=`${ESP}/_snapshot/${db_name}-${BACKUP_TIME}/${index}?wait_for_completion=true -H "Content-Type:application/json" -d '{"indices": "'${index}'","ignore_unavailable": true,"include_global_state": false}'`
echo "${SNAP_RESULT}" >> ${LOG_FILE}
fi
done
END_TIME=$(date "+%Y-%m-%d %H:%M:%S")
echo ">>>>Finish backup index : ${END_TIME}" >> ${LOG_FILE}
#step2
#删除过期的备仿
echo ">>>>Begin delete invalid backup index : ${CURRENT_TIME}" >> ${LOG_FILE}
DELETE_DATETIME=$(date +%Y%m%d -d "-3 day")
for dir in `ls -d /backup/zxb_backup/${db_name}-${DELETE_DATETIME}*`
do
rm -rf ${dir}
done
echo ">>>>Finish delete invalid backup index : ${END_TIME}" >> ${LOG_FILE}使用方式
vim database.txt
es1:192.168.248.231:9201:elastic:"xxx":"6.8.5"
es集群名:ip:port:用户名:密码:版本号
使用方法: sh backup_es_all_database.sh3.1.4 针对索引的备份脚本(backup_index.sh)
#!/bin/bash
#ES基于索引备份
#只要传入一个index.txt的索引列表即可备份
#Author:yuye-v1
#Date:2021/03/12
#Usage: sh backup_index.sh index.txt, index.txt为索引列表
USPD="${username}:${password}"
ES="http://${ip}:${port}"
BAK="/backup/elasticsearch"
LOG="/tmp/elasticsearch/logs"
function backup(){
echo "compressing $1 ..."
curl -u $USPD -H "Content-Type:application/json" -XPUT "$ES/_snapshot/$1" -d '{"type":"fs","settings":{"location":"'$BAK/$1'","compress":true}}'
echo "backup $1 ..."
curl -u $USPD -H "Content-Type:application/json" -XPUT "$ES/_snapshot/$1/$1?wait_for_completion=true" -d '{"indices": "'$1'","ignore_unavailable": true,"include_global_state": false}' > $LOG/$1.log
echo "backup $1 completed"
}
for index in $(cat $1)
do
backup $index
sleep 1s
done3.2 Elasticsearch恢复方案
3.2.1 注册备份片数据
curl -u elastic:paic1234A -H "Content-Type:application/json" -XPUT "http:// 192.168.248.231:9201/_snapshot/lcp-2020.09.24" -d '{"type":"fs","settings":{"location":"'/es-backup/master-backup/lcp-2020.09.24/lcp-2020.09.24'","max_restore_bytes_per_sec":"300mb","max_snapshot_bytes_per_sec":"300mb","compress":true}}'3.2.2 检查注册信息是否准确
curl -u elastic:xxx -H "Content-Type:application/json" -XGET "http:// 192.168.248.231:9201/_snapshot/lcp-2020.09.24/_all?pretty"3.2.3 恢复备份片
curl -u elastic:xxx -XPOST -H "Content-Type:application/json" "http:// 192.168.248.231:9201/_snapshot/lcp-2020.09.24/lcp-2020.09.24/_restore?wait_for_completion=true" -d '{"indices": "log_common_2020.09.24-000007","partial":true,"rename_pattern": "index_(.+)","rename_replacement": "log_common_2020.09.24-000007","ignore_unavailable":true,"ignore_index_settings":["index.routing.allocation.require.box_type"],"index_settings":{"index.number_of_replicas":0},"include_aliases":false}' &3.2.4 恢复脚本
#!/bin/bash
#Author: yuye-1
#Email: 13580480392@163.com
#Date 2020/12/18
#INDEX_NAME 为需要恢复的索引名字，BACKUP_NAME为备份片的名字,RESTORE_PATH为索引的恢复路径,RESTORE_IP为恢复主机ip,RESTORE_PORT为恢复主机端口
#Usage: sh restore_index.sh ${INDEX_NAME} ${BACKUP_NAME} ${RESTORE_PATH} ${RESTORE_IP} ${RESTORE_PORT}
INDEX_NAME=$1
BACKUP_NAME=$2
RESTORE_PATH=$3
RESTORE_IP=$4
RESTORE_PORT=$5
ES_USER="username"
ES_PASSWORD="password"
LOG_DIR=/tmp/es-recover
test -d ${LOG_DIR} || mkdir -p ${LOG_DIR}
### ---------------------------------------------------------------------------------------------------------------------------------------
#Registered snapshot
echo "Begin restore snapshot:" >> $${LOG_DIR}/${INDEX_NAME}_`date +%Y%m%d`.log
curl -u ${ES_USER}:${ES_PASSWORD} -H "Content-Type:application/json" -XPUT "http://${RESTORE_IP}:${RESTORE_PORT}/_snapshot/${BACKUP_NAME}" -d '{"type":"fs","settings":{"location":"'${RESTORE_PATH}'","max_restore_bytes_per_sec":"300mb","max_snapshot_bytes_per_sec":"300mb","compress":true}}'
if [ $? -ne 0 ];then
echo "[`date +"%F %T"`] ES registered failure ,please check" >> $${LOG_DIR}/${INDEX_NAME}_`date +%Y%m%d`.log
exit 1
fi
echo "Registered snapshot successfully!" >> $${LOG_DIR}/${INDEX_NAME}_`date +%Y%m%d`.log
#Check index's status
IS_EXISTS=`curl -u ${ES_USER}:${ES_PASSWORD} -H "Content-Type:application/json" -XGET "http://${RESTORE_IP}:${RESTORE_PORT}/_snapshot/${BACKUP_NAME}/_all?pretty"|grep ${INDEX_NAME}|wc -l`
if [ ${IS_EXISTS} -ne 1 ];then
echo "Index is not exists,please check" >> $${LOG_DIR}/${INDEX_NAME}_`date +%Y%m%d`.log
exit 1
fi
### ---------------------------------------------------------------------------------------------------------------------------------------
#Restore index
curl -u ${ES_USER}:${ES_PASSWORD} -XPOST -H "Content-Type:application/json" "http://${RESTORE_IP}:${RESTORE_PORT}/_snapshot/${BACKUP_NAME}/${BACKUP_NAME}/_restore?pretty" -d '{"indices":"'${INDEX_NAME}'","partial":true,"rename_pattern": "index_(.+)","rename_replacement": "'${INDEX_NAME}'","ignore_unavailable":true,"ignore_index_settings":["index.routing.allocation.require.box_type"]}'
if [ ${IS_EXISTS} -ne 1 ];then
echo "Index restore failed ,please check" >> $${LOG_DIR}/${INDEX_NAME}_`date +%Y%m%d`.log
exit 1
fi
echo "Restore index successfully!" >> $${LOG_DIR}/${INDEX_NAME}_`date +%Y%m%d`.log
echo "Index ${INDEX_NAME} restore end! " >> $${LOG_DIR}/${INDEX_NAME}_`date +%Y%m%d`.log4.注意事宜
- 1.需要建立统一的ES目录
- 2.ES的主机的username,uid,gid需要保持一致
- 3.ES备份的共享存储还可以通过sshfs挂载，下一篇文章我将介绍一下
5.参考文献
https://www.elastic.co/guide/en/elasticsearch/reference/6.8/modules-snapshots.html
备注： 如有疑问或者建议，请及时反馈13580480392@163.com。本人会及时反馈，感谢您的支持！
原创声明：本文系作者授权腾讯云开发者社区发表，未经许可，不得转载。
如有侵权，请联系 cloudcommunity@tencent.com 删除。
原创声明：本文系作者授权腾讯云开发者社区发表，未经许可，不得转载。
如有侵权，请联系 cloudcommunity@tencent.com 删除。
评论
登录后参与评论
推荐阅读
目录
相关产品与服务
Elasticsearch Service
腾讯云 Elasticsearch Service（ES）是云端全托管海量数据检索分析服务，拥有高性能自研内核，集成X-Pack。ES 支持通过自治索引、存算分离、集群巡检等特性轻松管理集群，也支持免运维、自动弹性、按需使用的 Serverless 模式。使用 ES 您可以高效构建信息检索、日志分析、运维监控等服务，它独特的向量检索还可助您构建基于语义、图像的AI深度应用。