linux 服务器安装s3cmd
yum install python-devel python-setuptools -y easy_install pip
报错： [root@localhost ~]# easy_install pip Searching for pip Reading http://pypi.python.org/simple/pip/ Couldn't find index page for 'pip' (maybe misspelled?) Scanning index of all packages (this may take a while) Reading http://pypi.python.org/simple/ No local packages or download links found for pip error: Could not find suitable distribution for Requirement.parse('pip')
原因是服务器没有安装pip命令 安装pip命令解决此报错（参考博主Linux下安装pip命令）
提示： 一开始cd /usr/lib/python2.6/site-packages/ [root@localhost site-packages]# cd S3 -bash: cd: S3: 没有那个文件或目录 tar xf s3cmd.tar.gz cd /root/s3cmd-1.6.0 pip install requests s3cmd==1.6.0
[root@localhost s3cmd-1.6.0]# pip install requests s3cmd==1.6.0
Downloading/unpacking requests
Downloading requests-2.18.4-py2.py3-none-any.whl (88kB): 88kB downloaded
Downloading/unpacking s3cmd==1.6.0
Downloading s3cmd-1.6.0.tar.gz (100kB): 100kB downloaded
Running setup.py (path:/tmp/pip_build_root/s3cmd/setup.py) egg_info for package s3cmd
Using xml.etree.ElementTree for XML processing
Downloading/unpacking certifi>=2017.4.17 (from requests)
Downloading certifi-2017.11.5-py2.py3-none-any.whl (330kB): 330kB downloaded
Downloading/unpacking idna>=2.5,<2.7 (from requests)
Downloading idna-2.6-py2.py3-none-any.whl (56kB): 56kB downloaded
Downloading/unpacking chardet>=3.0.2,<3.1.0 (from requests)
Downloading chardet-3.0.4-py2.py3-none-any.whl (133kB): 133kB downloaded
Downloading/unpacking urllib3>=1.21.1,<1.23 (from requests)
Downloading urllib3-1.22-py2.py3-none-any.whl (132kB): 132kB downloaded
Downloading/unpacking python-dateutil (from s3cmd==1.6.0)
Downloading python_dateutil-2.6.1-py2.py3-none-any.whl (194kB): 194kB downloaded
Downloading/unpacking python-magic (from s3cmd==1.6.0)
Downloading python_magic-0.4.15-py2.py3-none-any.whl
Downloading/unpacking six>=1.5 (from python-dateutil->s3cmd==1.6.0)
Downloading six-1.11.0-py2.py3-none-any.whl
Installing collected packages: requests, s3cmd, certifi, idna, chardet, urllib3, python-dateutil, python-magic, six
Running setup.py install for s3cmd
Using xml.etree.ElementTree for XML processing
changing mode of build/scripts-2.6/s3cmd from 644 to 755
changing mode of /usr/bin/s3cmd to 755
Successfully installed requests s3cmd certifi idna chardet urllib3 python-dateutil python-magic six
Cleaning up...cd /usr/lib/python2.6/site-packages/S3
[root@localhost site-packages]# cd S3/
[root@localhost S3]# ls
AccessLog.py BidirMap.py Config.py Crypto.py ExitCodes.py FileLists.py __init__.py PkgInfo.py S3.py SortedDict.py
AccessLog.pyc BidirMap.pyc Config.pyc Crypto.pyc ExitCodes.pyc FileLists.pyc __init__.pyc PkgInfo.pyc S3.pyc SortedDict.pyc
ACL.py CloudFront.py ConnMan.py Exceptions.py FileDict.py HashCache.py MultiPart.py Progress.py S3Uri.py Utils.py
ACL.pyc CloudFront.pyc ConnMan.pyc Exceptions.pyc FileDict.pyc HashCache.pyc MultiPart.pyc Progress.pyc S3Uri.pyc Utils.pycrz -y .s3cfg 上传密钥文件到服务器root目录下。
[root@localhost ~]# ls /root/.s3cfg /root/.s3cfg
ll /usr/lib/python2.6/site-packages/S3/Config.py cd /usr/lib/python2.6/site-packages/S3/ cp Config.py Config.py.bak
rz -y Config.py 上传Config.py 文件到服务器/usr/lib/python2.6/site-packages/S3/ 下 cp /root/s3cmd-1.6.0/s3cmd_mult /usr/local/sbin/ cd /usr/local/sbin/ chmod +x s3cmd_mult
同步到云存储的定时脚本
[root@localhost iplog]# cat /data/web/sync.sh#!/bin/bash
newdate=`date '+%Y-%m-%d'`
IP=`/sbin/ifconfig |grep 'inet addr' |awk -F":" '{print $2}' |awk -F' ' '{print $1}' |awk 'NR==1{print $1}' |awk -F'.' '{a=$3"."$4;print 'a'}'`
for i in `ls /data/log`
do
if [ "$i" != "$newdate" ];then
/usr/local/sbin/s3cmd_mult sync /data/log/"$i"/ s3://gaode/"$i"/cpc/"$IP"/ --no-check-md5 -P --parallel && rm -rf /data/log/$i >/dev/null 2>&1
fi
done把云端存储的日志文件目录以及目录下的文件拉取到服务器本地：
[root@localhost test001]# s3cmd sync s3://yuncongchu/2018-02-07/cpv/1.248/pv /opt/test001/ -v --no-check-md5
INFO: Retrieving list of remote files for s3://yuncongchu/2018-02-07/cpv/1.248/pv ...
INFO: Compiling list of local files...
INFO: Running stat() and reading/calculating MD5 values on 0 files, this may take some time...
INFO: Found 24 remote files, 0 local files
INFO: Verifying attributes...
INFO: Summary: 24 remote files to download, 0 local files to delete, 0 local files to hardlink
's3://yuncongchu/2018-02-07/cpv/1.248/pv/0.txt' -> '/opt/test001/pv/0.txt' [1 of 24]
76405655 of 76405655 100% in 43s 1713.03 kB/s done
's3://yuncongchu/2018-02-07/cpv/1.248/pv/1.txt' -> '/opt/test001/pv/1.txt' [2 of 24]
56033850 of 56033850 100% in 31s 1713.76 kB/s done
's3://yuncongchu/2018-02-07/cpv/1.248/pv/10.txt' -> '/opt/test001/pv/10.txt' [3 of 24]
19660800 of 80397458 24% in 11s 1703.68 kB/s^CSee ya!
[root@localhost test001]# ll /opt/test001/pv/*
-rw-r--r-- 1 apache apache 76405655 2月 7 23:59 /opt/test001/pv/0.txt
-rw-r--r-- 1 apache apache 56033850 2月 7 01:59 /opt/test001/pv/1.txt拉取云存储的上的单个日志文件到服务器本地：
[root@localhost test001]# s3cmd get s3://yuncongchu/2018-02-07/cpv/1.248/pv/9.txt ./9.txt
's3://yuncongchu/2018-02-07/cpv/1.248/pv/9.txt' -> './9.txt' [1 of 1]
75481869 of 75481869 100% in 43s 1707.12 kB/s done
[root@localhost test001]# s3cmd get s3://yuncongchu/2018-02-07/cpv/1.248/pv/8.txt ./
's3://yuncongchu/2018-02-07/cpv/1.248/pv/8.txt' -> './8.txt' [1 of 1]
76622153 of 76622153 100% in 42s 1779.85 kB/s done
[root@localhost test001]# ls
8.txt 9.txt pv查看云存储的上日志文件：
[root@localhost test001]# s3cmd ls s3://yuncongchu/2018-02-07/cpv/1.248/pv/*
2018-02-07 20:19 76405655 s3://yuncongchu/2018-02-07/cpv/1.248/pv/0.txt
2018-02-07 20:18 56033850 s3://yuncongchu/2018-02-07/cpv/1.248/pv/1.txt
[root@localhost test001]# s3cmd ls s3://yuncongchu/2018-02-07/cpv/1.248/*
DIR s3://yuncongchu/2018-02-07/cpv/1.248/count/
DIR s3://yuncongchu/2018-02-07/cpv/1.248/pv/