Ceph部署在Centos7上简明摘要

服务器全部安装centos7.5
192.168.3.8    ceph-admin（ceph-deploy）   mds1、mon1（也可以将monit节点另放一台机器）
192.168.3.7    ceph-node1                  osd1  
192.168.3.6    ceph-node2                  osd2 
-------------------------------------------------
每个节点修改主机名
# hostnamectl set-hostname ceph-admin
# hostnamectl set-hostname ceph-node1
# hostnamectl set-hostname ceph-node2
-------------------------------------------------
每个节点绑定主机名映射
# cat /etc/hosts
192.168.3.8    ceph-admin
192.168.3.7    ceph-node1 
192.168.3.6    ceph-node2
-------------------------------------------------
每个节点确认连通性
# ping -c 3 ceph-admin
# ping -c 3 ceph-node1
# ping -c 3 ceph-node2
-------------------------------------------------
每个节点关闭防火墙和selinux
# systemctl stop firewalld
# systemctl disable firewalld
# sed -i 's/SELINUX=enforcing/SELINUX=disabled/g' /etc/selinux/config
# setenforce 0
-------------------------------------------------
每个节点安装和配置NTP（官方推荐的是集群的所有节点全部安装并配置 NTP，需要保证各节点的系统时间一致。没有自己部署ntp服务器，就在线同步NTP）
# yum install ntp ntpdate ntp-doc -y
# systemctl restart ntpd
# systemctl status ntpd
-------------------------------------------------
每个节点准备yum源
删除默认的源，国外的比较慢
# yum clean all
# mkdir /mnt/bak
# mv /etc/yum.repos.d/* /mnt/bak/
 
下载阿里云的base源和epel源
# wget -O /etc/yum.repos.d/CentOS-Base.repo http://mirrors.aliyun.com/repo/Centos-7.repo
# wget -O /etc/yum.repos.d/epel.repo http://mirrors.aliyun.com/repo/epel-7.repo
 
添加ceph源
# vim /etc/yum.repos.d/ceph.repo
[ceph]
name=ceph
baseurl=http://mirrors.aliyun.com/ceph/rpm-jewel/el7/x86_64/
gpgcheck=0
priority =1
[ceph-noarch]
name=cephnoarch
baseurl=http://mirrors.aliyun.com/ceph/rpm-jewel/el7/noarch/
gpgcheck=0
priority =1
[ceph-source]
name=Ceph source packages
baseurl=http://mirrors.aliyun.com/ceph/rpm-jewel/el7/SRPMS
gpgcheck=0
priority=1
------------------------------------------------------------
每个节点创建cephuser用户，设置sudo权限
# useradd -d /home/cephuser -m cephuser
# echo "cephuser"|passwd --stdin cephuser
# echo "cephuser ALL = (root) NOPASSWD:ALL" | sudo tee /etc/sudoers.d/cephuser
# chmod 0440 /etc/sudoers.d/cephuser
# sed -i s'/Defaults requiretty/#Defaults requiretty'/g /etc/sudoers
 
测试cephuser的sudo权限
# su - cephuser
$ sudo su -
#
------------------------------------------------------------
配置相互间的ssh信任关系
现在ceph-admin节点上产生公私钥文件，然后将ceph-admin节点的.ssh目录拷贝给其他节点
[root@ceph-admin ~]# su - cephuser
[cephuser@ceph-admin ~]$ ssh-keygen -t rsa    #一路回车
[cephuser@ceph-admin ~]$ cd .ssh/
[cephuser@ceph-admin .ssh]$ ls
id_rsa  id_rsa.pub
[cephuser@ceph-admin .ssh]$ cp id_rsa.pub authorized_keys
 
[cephuser@ceph-admin .ssh]$ scp -r /home/cephuser/.ssh ceph-node1:/home/cephuser/
[cephuser@ceph-admin .ssh]$ scp -r /home/cephuser/.ssh ceph-node2:/home/cephuser/


 
然后在各节点直接验证cephuser用户下的ssh相互信任关系
$ ssh -p22 cephuser@ceph-admin
$ ssh -p22 cephuser@ceph-node1
$ ssh -p22 cephuser@ceph-node2

[root@ceph-admin ~]# su - cephuser
      
安装ceph-deploy
[cephuser@ceph-admin ~]$ sudo yum update -y && sudo yum install ceph-deploy -y
      
创建cluster目录
[cephuser@ceph-admin ~]$ mkdir cluster
[cephuser@ceph-admin ~]$ cd cluster/
[cephuser@ceph-admin cluster]$ ceph-deploy new ceph-admin ceph-node1 ceph-node2

[cephuser@ceph-admin cluster]$ vim ceph.conf 

将如下内容加入配置文件
osd pool default size = 2
mon_clock_drift_allowed = 1

[cephuser@ceph-admin cluster]$ ceph-deploy install --release=luminous ceph-admin ceph-node1 ceph-node2
      
初始化monit监控节点，并收集所有密钥
[cephuser@ceph-admin cluster]$ ceph-deploy mon create-initial
[cephuser@ceph-admin cluster]$ ceph-deploy gatherkeys ceph-admin
      
添加OSD到集群
检查OSD节点上所有可用的磁盘
[cephuser@ceph-admin cluster]$ ceph-deploy disk list ceph-node1 ceph-node2
[cephuser@ceph-admin ~]$ ssh ceph-node1 "sudo mkdir /var/local/osd0 && sudo chown ceph:ceph /var/local/osd0"
[cephuser@ceph-admin ~]$ ssh ceph-node2 "sudo mkdir /var/local/osd1 && sudo chown ceph:ceph /var/local/osd1"
[cephuser@ceph-admin ~]$ ceph-deploy osd prepare ceph-node1:/var/local/osd0 ceph-node2:/var/local/osd1
[cephuser@ceph-admin ~]$ ceph-deploy osd activate ceph-node1:/var/local/osd0 ceph-node2:/var/local/osd1

更新配置
[cephuser@ceph-admin ~]$ ceph-deploy admin ceph-admin ceph-node1 ceph-node2
[cephuser@ceph-admin ~]$ sudo chmod +r /etc/ceph/ceph.client.admin.keyring
[cephuser@ceph-admin ~]$ ssh ceph-node1 sudo chmod +r /etc/ceph/ceph.client.admin.keyring
[cephuser@ceph-admin ~]$ ssh ceph-node2 sudo chmod +r /etc/ceph/ceph.client.admin.keyring

服务状态
[cephuser@ceph-admin ~]$ ceph health
HEALTH_OK

[cephuser@ceph-admin ~]$ ceph -s
    cluster 50fcb154-c784-498e-9045-83838bc3018e
     health HEALTH_OK
     monmap e1: 3 mons at {ceph-admin=192.16.3.8:6789/0,ceph-node1=192.16.3.7:6789/0,ceph-node2=192.16.3.6:6789/0}
            election epoch 4, quorum 0,1,2 ceph-node2,ceph-node1,ceph-admin
     osdmap e12: 2 osds: 2 up, 2 in
            flags sortbitwise,require_jewel_osds
      pgmap v624: 64 pgs, 1 pools, 136 MB data, 47 objects
            20113 MB used, 82236 MB / 102350 MB avail
                  64 active+clean

挂载测试                 
[cephuser@ceph-admin ~]$ sudo rbd create foo --size 4096 -m ceph-node1
[cephuser@ceph-admin ~]$ sudo rbd map foo --pool rbd --name client.admin -m ceph-node1
[cephuser@ceph-admin ~]$ sudo mkfs.ext4 -m0 /dev/rbd/rbd/foo
[cephuser@ceph-admin ~]$ sudo mkdir /mnt/ceph-block-device
[cephuser@ceph-admin ~]$ sudo mount /dev/rbd/rbd/foo /mnt/ceph-block-device