ceph对象存储折腾记原

domain0

发布于 2018-08-02 11:12:42

5900

发布于 2018-08-02 11:12:42

###前言一直想弄对象存储，以前弄过一次，不是很理解region是个什么东西，后来时间和工作上的原因没有再折腾，这两天闲了下来，再次折腾了一次。我是参考的ceph的中文翻译文挡进行的部署和测试。传送门,文档里面介绍的和ceph本身的版本存在脱节的现象，可能初次接触的人会因为服务启动的问题摸不着头脑。 ###关于部署安装ceph必要的软件包，配置好公共密钥和ceph mon的配置，这里我不再谈了。对象存储额外需要安装的包是：ceph-radosgw和ceph-common 安装完毕你的系统上应该至少存在三个命令：rados 、 radosgw 、 radosgw-admin 其中整个对象网关服务就是由radosgw来启动的，radosgw-admin负责管理对象资源（用户，权限，bucket），rados基本算一个比较简单的s3客户端（？我这里可能理解不是很精确） ####配置 ceph.conf

[global]
fsid = xxxxxxxxxxxxxxxxxxxxxxxxxxxx
mon_initial_members = t41,t42,t45
mon_host = 192.168.168.41,192.168.168.42,192.168.168.45
auth_cluster_required = cephx
auth_service_required = cephx
auth_client_required = cephx

# t56是服务器的hostname,由hostname -s命令可获取
[client.radosgw.t56]
host = your_ceph_rados_host
keyring = /etc/ceph/ceph.client.radosgw.keyring
rgw_socket_path = "/var/run/ceph/ceph.radosgw.gateway.fastcgi.sock"
log_file = "/data/logs/client.radosgw.gateway.log"
rgw_frontends = civetweb port=80
rgw_print_continue = true

所有配置选项参考传送门 ####密钥环

#创建方式
ceph auth create client.radosgw.t56 osd 'allow rwx' mon 'allow rwx' -o /etc/ceph/ceph.client.radosgw.keyring
#密钥环的样子
[client.radosgw.t56]
	key = xxxxxxxxxxxxxxxxxxxxxxxx==

####服务的启动方式

#有三种启动方式
1. /etc/init.d/ceph-radosgw  [start|stop|status|reload]
2. systemctl start  ceph-radosgw
3. radosgw  -c /etc/ceph/ceph.conf -n client.radosgw.t56

本质都是第三种启动，无非写了个脚本而已。radosgw -h可以看看其他的参数，其中-f前台执行和--debug_ms设置调试等级有利于调试。 ####授权用户

[root@t56 /data]# radosgw-admin user create --uid="xueyi28" --display-name="Xueyi"
{
    "user_id": "xueyi28",
    "display_name": "Xueyi",
    "email": "",
    "suspended": 0,
    "max_buckets": 1000,
    "auid": 0,
    "subusers": [],
    "keys": [
        {
            "user": "xueyi28",
            "access_key": "xxxxxxxxxxxxxxx",
            "secret_key": "xxxxxxxxxxxxxxxxxxxxxxxxxxxxxx"
        }
    ],
    "swift_keys": [],
    "caps": [],
    "op_mask": "read, write, delete",
    "default_placement": "",
    "placement_tags": [],
    "bucket_quota": {
        "enabled": false,
        "max_size_kb": -1,
        "max_objects": -1
    },
    "user_quota": {
        "enabled": false,
        "max_size_kb": -1,
        "max_objects": -1
    },
    "temp_url_keys": []
}

这里有一个要注意的，这数据格式是json格式，access_key和secret_key上可能存在反斜杠之类的转意字符，用key的时候，注意把转意字符处理一下，省的纠结生成的key老是验证不过。

####关于region 对象存储设计考虑到数据中心区域的问题，这里的region就是区域的标识。比如中国西部数据中心，中国南方数据中心，不同的区域数据中心的bucket还可以异地同步（？下一步需要研究的），配合用户的DNS，可以让不同区域的用户连接不同区域的数据中心数据。

[root@t56 /data]# radosgw-admin region get
{
    "name": "default",
    "api_name": "",
    "is_master": "true",
    "endpoints": [],
    "hostnames": [],
    "master_zone": "",
    "zones": [
        {
            "name": "default",
            "endpoints": [],
            "log_meta": "false",
            "log_data": "false",
            "bucket_index_max_shards": 0
        }
    ],
    "placement_targets": [
        {
            "name": "default-placement",
            "tags": []
        }
    ],
    "default_placement": "default-placement"
}

[root@t56 /data]#  radosgw-admin regions list 
{
    "default_info": {
        "default_region": "default"
    },
    "regions": [
        "default"
    ]
}

我这个是测试用的，就只有一个默认的default region。

###数据的读写和bucket的使用我这里主要说php sdk使用s3接口。ceph给的文档里面的大多数sdk版本都是上个世纪的，亚马逊的s3 php sdk变的乱七八糟，入门比较慢，搞半天也搞不明白。我留了一个老版本的php sdk，凑合能接到ceph的文档。测试代码

<\?php
define('AWS_KEY', 'your_access_key');
define('AWS_SECRET_KEY', 'your_secret_key');
define('AWS_CANONICAL_ID', 'xueyi28');
define('AWS_CANONICAL_NAME', 'Xueyi');
$HOST = 'your_ceph_radosgw_host';

// require the amazon sdk for php library
require_once 'AWSSDKforPHP/sdk.class.php';

// Instantiate the S3 class and point it at the desired host
$Connection = new AmazonS3(array(
        'key' => AWS_KEY,
        'secret' => AWS_SECRET_KEY,
        'canonical_id' => AWS_CANONICAL_ID,
        'canonical_name' => AWS_CANONICAL_NAME,
));
$Connection->use_ssl = false; //禁用ssl
$Connection->set_hostname($HOST);
$Connection->enable_path_style(true); //采用path的模式，不然就是域名模式，bucket会成为根域名的子域名
$Connection->allow_hostname_override(false);
$Connection->path_style = true; //sdk哪里有点问题，path_style配置老是不生效，可以调一下sdk的代码
#$Connection->create_bucket('my-new-bucket', AmazonS3::REGION_US_E1); //region可以在代码中进行定制，一个region对应一个地方域名。我们是default region 就对应REGION_US_E1，默认的就这个，看代码就明白了
$ListResponse = $Connection->list_buckets();
$Buckets = $ListResponse->body->Buckets->Bucket;
foreach ($Buckets as $Bucket) {
        echo $Bucket->Name . "\t" . $Bucket->CreationDate . "\n";
}
$Connection->create_object('my-new-bucket', 'hello.txt', array(
        'body' => "Hello World!",
));
$Connection->set_object_acl('my-new-bucket', 'hello.txt', AmazonS3::ACL_PUBLIC);

php sdk传送门核心逻辑代码/services/s3.class.php

###终端玩转对象存储在终端下要用s3对象存储，最好的工具无非是s3cmd,一般yum就可以直接安装下来，不过这玩意要注意版本，老版本的和最新的文档用法差别比较大。 ####首先要定制一下s3cmd的配置

[root@t56 ~]# s3cmd --configure 

Enter new values or accept defaults in brackets with Enter.
Refer to user manual for detailed description of all options.

Access key and Secret key are your identifiers for Amazon S3. Leave them empty for using the env variables.
Access Key: xxxxxxxxxxxxxxxxxxxxxxxx
Secret Key: xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
Default Region [US]: default

Encryption password is used to protect your files from reading
by unauthorized persons while in transfer to S3
Encryption password: 
Path to GPG program [/usr/bin/gpg]: 

When using secure HTTPS protocol all communication with Amazon S3
servers is protected from 3rd party eavesdropping. This method is
slower than plain HTTP, and can only be proxied with Python 2.7 or newer
Use HTTPS protocol [No]: No

On some networks all internet access must go through a HTTP proxy.
Try setting it here if you can't connect to S3 directly
HTTP Proxy server name: 

New settings:
  Access Key: xxxxxxxxxxxxxxxxxxxxxxxxxx
  Secret Key: xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
  Default Region: default
  Encryption password: 
  Path to GPG program: /usr/bin/gpg
  Use HTTPS protocol: False
  HTTP Proxy server name: 
  HTTP Proxy server port: 0

Test access with supplied credentials? [Y/n] n

Save settings? [y/N] y
Configuration saved to '/root/.s3cfg'

定制完之后要再编辑一下/root/.s3cfg文件，把你自定义的Host写进去，定义好你的Host和region、bucket的访问方式给一个例子

[root@t56 ~]# cat .s3cfg 
[default]
access_key = xxxxxxxxxxxxxxxxxxxxxxx
access_token = 
add_encoding_exts = 
add_headers = 
bucket_location = default
ca_certs_file = 
cache_file = 
check_ssl_certificate = True
cloudfront_host = your_ceph_rgw_host
default_mime_type = binary/octet-stream
delay_updates = False
delete_after = False
delete_after_fetch = False
delete_removed = False
dry_run = False
enable_multipart = True
encoding = ANSI_X3.4-1968
encrypt = False
expiry_date = 
expiry_days = 
expiry_prefix = 
follow_symlinks = False
force = False
get_continue = False
gpg_command = /usr/bin/gpg
gpg_decrypt = %(gpg_command)s -d --verbose --no-use-agent --batch --yes --passphrase-fd %(passphrase_fd)s -o %(output_file)s %(input_file)s
gpg_encrypt = %(gpg_command)s -c --verbose --no-use-agent --batch --yes --passphrase-fd %(passphrase_fd)s -o %(output_file)s %(input_file)s
gpg_passphrase = 
guess_mime_type = True
host_base = your_ceph_rgw_host
host_bucket = your_ceph_rgw_host/%(bucket)
human_readable_sizes = False
ignore_failed_copy = False
invalidate_default_index_on_cf = False
invalidate_default_index_root_on_cf = True
invalidate_on_cf = False
list_md5 = False
log_target_prefix = 
max_delete = -1
mime_type = 
multipart_chunk_size_mb = 15
preserve_attrs = True
progress_meter = True
proxy_host = 
proxy_port = 0
put_continue = False
recursive = False
recv_chunk = 4096
reduced_redundancy = False
restore_days = 1
secret_key = xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
send_chunk = 4096
server_side_encryption = False
signature_v2 = False
simpledb_host = sdb.your_ceph_rgw_host
skip_existing = False
socket_timeout = 300
urlencoding_mode = normal
use_https = False
use_mime_magic = True
verbosity = WARNING
website_endpoint = http://%(bucket)s.s3-website-%(location)your_ceph_rgw_host/
website_error = 
website_index = index.html

有几个地方修改你得按照你自己的需求去整，需要什么就整什么，让我们试试

[root@t56 ~]# s3cmd ls
2016-11-23 12:36  s3://my-new-bucket
[root@t56 ~]# s3cmd  ls s3://my-new-bucket 
2016-11-23 12:38        12   s3://my-new-bucket/hello.txt
[root@t56 ~]# s3cmd put test.log  s3://my-new-bucket/test.log
test.log -> s3://my-new-bucket/test.log  [1 of 1]
 242 of 242   100% in    0s     3.48 kB/s  done
[root@t56 ~]# s3cmd setacl s3://my-new-bucket/test.log --acl-public 
s3://my-new-bucket/test.log: ACL set to Public  [1 of 1]
[root@t56 ~]# s3cmd del s3://my-new-bucket/hello.txt 
File s3://my-new-bucket/hello.txt deleted

//递归授权的方式 s3cmd setacl s3://myexamplebucket.calvium.com/ --acl-public --recursive

棒极了

本文参与腾讯云自媒体分享计划，分享自作者个人站点/博客。

原始发表：2016/11/24 ，如有侵权请联系 cloudcommunity@tencent.com 删除

其他

本文分享自作者个人站点/博客前往查看

如有侵权，请联系 cloudcommunity@tencent.com 删除。

本文参与腾讯云自媒体分享计划，欢迎热爱写作的你一起参与！

其他

登录后参与评论

0 条评论

热度