ceph-Luminous对ipv6的支持

针对ceph Luminous over ipv6做了一些调研，基本功能测试

需求 随着ipv6使用得越来越广，很多网络设施逐步地需要支持ipv6，而ceph作为可大规模部署的分布式存储系统，ipv6的支持是必选的，本文主要介绍ceph over ipv6的场景及其功能使用

环境情况 测试环境，一个ceph集群加一台主机进行功能验证

ceph环境：ceph version 12.2.11 luminous (stable) 客户端机器：CentOS Linux release 7.5.1804 (Core) ceph集群每个节点两个网卡，均配置ipv6地址

集群情况 简单起见，配置不多 [cephfsd@ceph1 ~]$ sudo ceph -s cluster:

id:     db45806c-b322-450d-8f8a-3c07cdcd0b8e
health: HEALTH_OK

services:

mon: 3 daemons, quorum ceph1,ceph2,ceph3
mgr: ceph1(active), standbys: ceph2
mds: firstcephfs-1/1/1 up  {0=ceph1=up:active}, 1 up:standby
osd: 3 osds: 3 up, 3 in
rgw: 3 daemons active

data:

pools:   8 pools, 176 pgs
objects: 2.11k objects, 18.5MiB
usage:   3.41GiB used, 20.6GiB / 24.0GiB avail
pgs:     176 active+clean

使用ceph-deploy部署的，配置文件大部分都是自动生成的，如果手工部署，需要注意ms_bind_ipv6和rgw_frontends这两个参数的配置 [global] fsid = db45806c-b322-450d-8f8a-3c07cdcd0b8e ms_bind_ipv6 = true mon_initial_members = ceph1, ceph2, ceph3 mon_host = [2001:470:18:ac4::2],[2001:470:18:ac4::3],[2001:470:18:ac4::4] auth_cluster_required = cephx auth_service_required = cephx auth_client_required = cephx

public_network = 2001:470:18:ac4::2/64 cluster_network = 2002:470:18:ac4::2/64

rgw_frontends = "civetweb port=[::]:7480"

客户端功能验证 事实上，只要集群起来跑得没问题，功能应该是没问题的，有几个细节地方记录一下

对象存储功能 使用python的boto包进行测试，这里注意endpoint的指定方式，如果不带[]会有问题

!/usr/bin/env python

import boto import time import boto.s3.connection

access_key = 'aaaaaa' secret_key = 'bbbbbbbbb' endpoint = '[2001:470:18:ac4::2]' conn = boto.connect_s3(aws_access_key_id = access_key,

                   aws_secret_access_key = secret_key,
                   host = endpoint,
                   port = 7480,
                   is_secure=False,
                   calling_format = boto.s3.connection.OrdinaryCallingFormat())

conn.create_bucket('my-bucket3') my_bucket = conn.get_bucket('my-bucket3') newobj = my_bucket.new_key(str(time.time())+'.rst') newobj.set_contents_from_filename('README.rst') conn.close() 另外，s3cmd对接ipv6的时候，也是需要将ipv6部分用[]起来，否则会报错

[tanweijie@openattic ~]$ cat .s3cfg |grep 'host_' host_base = [2001:470:18:ac4::2]:7480 host_bucket = [2001:470:18:ac4::2]:7480/%(bucket)

CEPHFS功能 在crushmap中禁用掉chooseleaf_vary_r和chooseleaf_stable后，在测试设备上挂载cephfs，写入文件、读取文件均无问题，要注意的是挂载的方式，ipv6地址依然需要[]起来

[root@openattic tanweijie]# df -h Filesystem Size Used Avail Use% Mounted on /dev/mapper/centos-root 17G 2.3G 15G 14% / devtmpfs 484M 0 484M 0% /dev tmpfs 496M 0 496M 0% /dev/shm tmpfs 496M 6.8M 490M 2% /run tmpfs 496M 0 496M 0% /sys/fs/cgroup /dev/sda1 1014M 130M 885M 13% /boot tmpfs 100M 0 100M 0% /run/user/0 tmpfs 100M 0 100M 0% /run/user/1001

[root@openattic tanweijie]# cp -r openattic /media/ [root@openattic tanweijie]# cd /media [root@openattic media]# ls openattic [root@openattic media]# rm -rf openattic [root@openattic media]# ls [root@openattic media]#