专栏首页生信宝典Linux服务器数据定期同步和备份方式

Linux服务器数据定期同步和备份方式

数据安全是做数据分析的人需要关注的一大问题。对于我们分析的关键数据、使用的关键脚本都需要定期备份。

scp

最简单的备份方式,就是使用cp (本地硬盘)或scp (远程硬盘)命令,给自己的结果文件新建一个拷贝;每有更新,再拷贝一份。具体命令如下:

cp -fur source_project project_bak
scp -r source_project user@remote_server_ip:project_bak

为了实现定期备份,我们可以把上述命令写入crontab程序中,设置每天的晚上23:00执行。对于远程服务器的备份,我们可以配置免密码登录,便于自动备份。后台输入免密码登录服务器,获取免密码登录服务器的方法。

# Crontab format
# Minute  Hour  Day  Month  Week  command 
# * 表示每分/时/天/月/周
# 每天23:00 执行cp命令
0          23      *       *       *      cp -fur source_project project_bak
# */2 表示每隔2分分/时/天/月/周执行命令
# 每隔24小时执行cp命令
0          */24      *       *       *      cp -fur source_project project_bak
0          0          */1     *        *     scp -r source_project user@remote_server_ip:project_bak

# 另外crotab还有个特殊的时间
# @reboot: 开机运行指定命令
@reboot cmd

rsync

cpscp使用简单,但每次执行都会对所有文件进行拷贝,耗时耗力,尤其是需要拷贝的内容很多时,重复拷贝对时间和硬盘都是个损耗。

rsync则是一个增量备份工具,只针对修改过的文件修改过的部分进行同步备份,大大缩短了传输的文件的数量和传输时间。具体使用如下 :

# 把本地project目录下的东西备份到远程服务器的/backup/project目录下
# 注意第一个project后面的反斜线,表示拷贝目录内的内容,不在目标目录新建project文件夹。注意与第二个命令的比较,两者实现同样的功能。
# -a: archive mode, quals -rlptgoD
# -r: 递归同步
# -p: 同步时保留原文件的权限设置
# -u: 若文件在远端做过更新,则不同步,避免覆盖远端的修改
# -L: 同步符号链接链接的文件,防止在远程服务器出现文件路径等不匹配导致的软连接失效
# -t: 保留修改时间
# -v: 显示更新信息
# -z: 传输过程中压缩文件,对于传输速度慢时适用
rsync -aruLptvz --delete project/ user@remoteServer:/backup/project
rsync -aruLptvz --delete project user@remoteServer:/backup/

rsync所做的工作为镜像,保证远端服务器与本地文件的统一。如果本地文件没问题,远端也不会有问题。但如果发生误删或因程序运行错误,导致文件出问题,而在同步之前又没有意识到的话,远端的备份也就没了备份的意义,因为它也被损坏了。误删是比较容易发现的,可以及时矫正。但程序运行出问题,则不一定了。

rdiff-backup

这里推荐一个工具rdiff-backup不只可以做增量备份,而且会保留每次备份的状态,新备份和上一次备份的差别,可以轻松回到之前的某个版本。唯一的要求就是,本地服务器和远端服务器需要安装统一版本的rdiff-backup。另外还有2款工具 duplicity`Rsnapshot也可以做类似工作,但方法不一样,占用的磁盘空间也不一样,具体可查看原文链接中的比较。

具体的rdiff-backup安装和使用如下 (之前写的是英文,内容比较简单,就不再翻译了):

Install rdiff-backup at both local and remote computers

  • Install requirements #install for ubuntu, debian sudo apt-get install python-dev librsync-dev #self compile #downlaod rsync-dev from https://sourceforge.net/project/showfiles.php?group_id=56125 tar xvzf librsync-0.9.7.tar.gz export CFLAGS="$CFLAGS -fPIC" ./configure --prefix=/home/user/rsync --with-pic make make install
  • Install rdiff-backup #See Reference part for download link # http://www.nongnu.org/rdiff-backup/ python setup.py install --prefix=/home/user/rdiff-backup #If you complied rsync-dev yourself, please specify the location of rsync-dev python setup.py --librsync-dir=/home/user/rsync install -- prefix=/home/user/rdiff-backup
  • Add exeutable files and python modules to environmental variables #Add the following words into .bashrc or .bash_profile or any other config files export PATH=${PATH}:/home/user/rdiff-backup/bin export PYTHONPATH=${PYTHONPATH}:/home/user/rdiff-backup/lib/python2.x/site-packages #pay attention to the x in python2.x of above line which can be 6 or 7 depending on #the Python version used.
  • Test environmental variable when executing commands through ssh ssh user@host 'echo ${PATH}' #When I run this command in my local computer, #I found only system environmetal variable is used #and none of my self-defined environmetal variable is used. #Then, I modified the following lines in file 'SetConnections.py' in #/home/user/rdiff-backup/lib/python2.x/site-packages/rdiff_backup #to set environmental explicitly when login. #pay attention to the single quote used inside double quote __cmd_schema = "ssh -C %s 'source ~/.bash_profile; rdiff-backup --server'" __cmd_schema_no_compress = "ssh %s 'source ~/.bash_profile; rdiff-backup --server'" #choose the one contains environmental variable for rdiff-backup from .bash_profile and .bashrc.

Use rdiff-backup

  • Start backup
    • rdiff-backup --no-compression --print-statistics user@host::/home/user/source_dir destination_dir
    • If the destination_dir exists, please add --force like rdiff-backup --no-compression --force --print-statistics user@host::/home/user/source_dir destination_dir. All things in original destination_dir will be depleted.
    • If you want to exclude or include special files or dirs please specify like --exclude '**trash' or --include /home/user/source_dir/important.
  • Timely backup your data
    • Add the above command into crontab (hit 'crontab -e' in terminal to open crontab) in the format like 5 22 */1 * * command which means executing the command at 22:05 everyday.
  • Restore data
    • Restore the latest data by running rdiff-backup -r now destination_dir user@host::/home/user/source_dir.restore. Add --force if you want to restore to source_dir.
    • Restore files 10 days ago by running rdiff-backup -r 10D destination_dir user@host::/home/user/source_dir.restore. Other acceptable time formats include 5m4s (5 minutes 4 seconds) and 2014-01-01 (January 1st, 2014).
    • Restore files from an increment file by running rdiff-backup destination_dir/rdiff-backup-data/increments/server_add.2014-02-21T09:22:45+08:00.missing user@host::/home/user/source_dir.restore/server_add. Increment files are stored in destination_dir/rdiff-backup-data/increments/server_add.2014-02-21T09:22:45+08:00.missing.
  • Remove older records to save space
    • Deletes all information concerning file versions which have not been current for 2 weeks by running rdiff-backup --remove-older-than 2W --force destination_dir. Note that an existing file which has not changed for a year will still be preserved. But a file which was deleted 15 days ago can not be restored after this command. Normally one should use --force since it is used to delete multiple increments at the same time which --remove-older-than refuses to do by default.
    • Only keeps the last n rdiff-backup sessions by running rdiff-backup --remove-older-than 20B --force destination_dir.
  • Statistics
    • Lists increments in given golder by rdiff-backup --list-increments destination_dir/.
    • Lists of files changed in last 5 days by rdiff-backup --list-changed-since 5D destination_dir/.
    • Compare the difference between source and bak by rdiff-backup --compare user@host::source-dir destination_dir
    • Compare the sifference between source and bak (as it was two weeks ago) by rdiff-backup --compare-at-time 2W user@host::source-dir destination_dir.

A complete script (automatically sync using crontab)

#!/bin/bash

export PYTHONPATH=${PYTHONPATH}:/soft/rdiff_backup/lib/python2.7/site-packages/

rdiff-backup --no-compression -v5 --exclude '**trash' user@server::source/ bak_dir/

ret=$?
if test $ret -ne 0; then
    echo "Wrong in bak" | mutt -s "Wrong in bak" bak@mail.com
else
    echo "Right in bak" | mutt -s "Right in bak" bak@mail.com
fi

echo "Finish rdiff-backup $0 ---`date`---"  >>bak.log 2>&1

echo "`rdiff-backup --exclude '**trash' --compare-at-time 1D user@server::source/ bak_dir/`" | mutt -s "Lists of baked files" bak@mail.com

References

  • rdiff-backup
  • duplicity
  • rsnapshot
  • http://www.saltycrane.com/blog/2008/02/backup-on-linux-rsnapshot-vs-rdiff/
  • http://james.lab6.com/2008/07/09/rdiff-backup-and-duplicity/
  • http://bitflop.com/document/75
  • http://askubuntu.com/questions/2596/comparison-of-backup-tools
  • http://www.reddit.com/r/linux/comments/fgmbb/rdiffbackup_duplicity_or_rsnapshot_which_is/
  • http://serverfault.com/questions/491341/optimize-space-rdiff-backup
  • Another great post on usage of rdiff-backup

本文分享自微信公众号 - 生信宝典(Bio_data),作者:陈同

原文出处及转载信息见文内详细说明,如有侵权,请联系 yunjia_community@tencent.com 删除。

原始发表时间:2017-06-04

本文参与腾讯云自媒体分享计划,欢迎正在阅读的你也加入,一起分享。

我来说两句

0 条评论
登录 后参与评论

相关文章

  • 为啥我的Python这么慢 - 项查找 (二)

    上一篇为啥我的Python这么慢, 字符串的加和和join被陈群主分享到biopython-生信QQ群时,乐平指出字典的写法存在问题,并给了一篇知乎的链接htt...

    生信宝典
  • NBT:单细胞转录组新降维可视化方法PHATE

    NGS系列文章包括NGS基础、转录组分析 (Nature重磅综述|关于RNA-seq你想知道的全在这)、ChIP-seq分析 (ChIP-seq基本分析流程)、...

    生信宝典
  • 简单可视化-送你一双发现美的眼睛

    用PyMOL展示配体和受体相互作用的原子和氢键 为了简化展示过程,我们设计了一个pml脚本 (脚本内有很详细的解释),只需要修改脚本里面受体和配体的名字,然后在...

    生信宝典
  • 一周简报|Udesk智能客服英文版正式上线,成为国内智能客服第一家

    编辑导语 Udesk智能客服英文版正式上线,成为国内智能客服第一家;PPMessage开源Android和iOS开发SDK;智齿客服助力航信集团,近40家分支机...

    BestSDK
  • Android StrictMode运行流程(推荐)

    strictmode是android在 API9后引入的检测影响app运行流畅性的一种机制,例如我们都知道的主线程中不允许有网络操作这条规则就是严苛模式规则的一...

    砸漏
  • 仅用CPU就能跑到1000FPS,这是开源的C++跨平台人脸检测项目

    项目地址:https://github.com/ShiqiYu/libfacedetection

    机器之心
  • Android开发:Android Studio运行模拟器报错AVD Nex... is already running解决办法

    前段时间在Android开发的时候,遇到Android Studio运行模拟器的时候提示AVD Nexus_5X_API_26 is already runni...

    三掌柜
  • Android模拟器介绍(一)

    在前面中已经介绍了Android环境的搭建,本文这里不再重复。本文中,仅介绍Android模拟器的使用以及安装,配置。

    无涯WuYa
  • 数学--数论--POJ281(线性同余方程)

    埃琳娜(Elina)正在阅读刘如家(Rujia Liu)写的书,其中介绍了一种表达非负整数的奇怪方法。方式描述如下: 选择k个不同的正整数a 1,a 2,…,...

    风骨散人Chiam
  • 算法分类 ,时间复杂度 ,空间复杂度,优

      今天给大家带来一篇关于算法排序的分类,算法的时间复杂度,空间复杂度,还有怎么去优化算法的文章,喜欢的话,可以关注,有什么问题,可以评论区提问,可以与我私信...

    py3study

扫码关注云+社区

领取腾讯云代金券