如何从 Ceph (Luminous) 集群中安全移除 OSD

OSD.png

工作中需要从 Ceph 的集群中移除一台存储服务器,挪作他用。Ceph 存储空间即使在移除该存储服务器后依旧够用,所以操作是可行的,但集群已经运行了很长时间,每个服务器上都存储了很多数据,在数据无损的情况下移除,看起来也不简单。

1. OSD 布局

先来看看 OSD 的布局

$ ceph osd tree
ID CLASS WEIGHT    TYPE NAME      STATUS REWEIGHT PRI-AFF 
-1       265.25757 root default                           
-5       132.62878     host osd7                          
24   hdd   5.52620         osd.24     up  1.00000 1.00000 
25   hdd   5.52620         osd.25     up  1.00000 1.00000 
26   hdd   5.52620         osd.26     up  1.00000 1.00000 
27   hdd   5.52620         osd.27     up  1.00000 1.00000 
28   hdd   5.52620         osd.28     up  1.00000 1.00000 
29   hdd   5.52620         osd.29     up  1.00000 1.00000 
30   hdd   5.52620         osd.30     up  1.00000 1.00000 
31   hdd   5.52620         osd.31     up  1.00000 1.00000 
32   hdd   5.52620         osd.32     up  1.00000 1.00000 
33   hdd   5.52620         osd.33     up  1.00000 1.00000 
34   hdd   5.52620         osd.34     up  1.00000 1.00000 
35   hdd   5.52620         osd.35     up  1.00000 1.00000 
36   hdd   5.52620         osd.36     up  1.00000 1.00000 
37   hdd   5.52620         osd.37     up  1.00000 1.00000 
38   hdd   5.52620         osd.38     up  1.00000 1.00000 
39   hdd   5.52620         osd.39     up  1.00000 1.00000 
40   hdd   5.52620         osd.40     up  1.00000 1.00000 
41   hdd   5.52620         osd.41     up  1.00000 1.00000 
42   hdd   5.52620         osd.42     up  1.00000 1.00000 
43   hdd   5.52620         osd.43     up  1.00000 1.00000 
44   hdd   5.52620         osd.44     up  1.00000 1.00000 
45   hdd   5.52620         osd.45     up  1.00000 1.00000 
46   hdd   5.52620         osd.46     up  1.00000 1.00000 
47   hdd   5.52620         osd.47     up  1.00000 1.00000 
-3       132.62878     host osd8                          
 0   hdd   5.52620         osd.0      up  1.00000 1.00000 
 1   hdd   5.52620         osd.1      up  1.00000 1.00000 
 2   hdd   5.52620         osd.2      up  1.00000 1.00000 
 3   hdd   5.52620         osd.3      up  1.00000 1.00000 
 4   hdd   5.52620         osd.4      up  1.00000 1.00000 
 5   hdd   5.52620         osd.5      up  1.00000 1.00000 
 6   hdd   5.52620         osd.6      up  1.00000 1.00000 
 7   hdd   5.52620         osd.7      up  1.00000 1.00000 
 8   hdd   5.52620         osd.8      up  1.00000 1.00000 
 9   hdd   5.52620         osd.9      up  1.00000 1.00000 
10   hdd   5.52620         osd.10     up  1.00000 1.00000 
11   hdd   5.52620         osd.11     up  1.00000 1.00000 
12   hdd   5.52620         osd.12     up  1.00000 1.00000 
13   hdd   5.52620         osd.13     up  1.00000 1.00000 
14   hdd   5.52620         osd.14     up  1.00000 1.00000 
15   hdd   5.52620         osd.15     up  1.00000 1.00000 
16   hdd   5.52620         osd.16     up  1.00000 1.00000 
17   hdd   5.52620         osd.17     up  1.00000 1.00000 
18   hdd   5.52620         osd.18     up  1.00000 1.00000 
19   hdd   5.52620         osd.19     up  1.00000 1.00000 
20   hdd   5.52620         osd.20     up  1.00000 1.00000 
21   hdd   5.52620         osd.21     up  1.00000 1.00000 
22   hdd   5.52620         osd.22     up  1.00000 1.00000 
23   hdd   5.52620         osd.23     up  1.00000 1.00000 

一共两台服务器,48 个 OSD。需要把 osd8 移除,那么就需要把上面的所有的 24 个 OSD 全部删除。

2. 单个 OSD 进程删除流程

以移除 osd.0 为例看一下移除 OSD 的流程:

2.1 将状态设置成 out

首先要现将 OSD 状态设置成 out。

$ ceph osd out 0
marked out osd.0. 

这个阶段 ceph 会自动将处于 out 状态 OSD 中的数据迁移到其他状态正常的 OSD 上,所以在执行完成后,需要使用 ceph -w 查看数据迁移流程。等到不再有输出后,数据迁移完毕。

$ ceph -w  
  cluster:
    id:     063ed8d6-fc89-4fcb-8811-ff23915983e7
    health: HEALTH_ERR
            12408/606262 objects misplaced (2.047%)
            6 scrub errors
            Reduced data availability: 2 pgs peering
            Possible data damage: 5 pgs inconsistent
            application not enabled on 7 pool(s)
 
  services:
    mon: 3 daemons, quorum dell1,dell2,dell3
    mgr: dell1(active)
    mds: cephfs-1/1/1 up  {0=dell1=up:active}, 2 up:standby
    osd: 48 osds: 48 up, 47 in; 44 remapped pgs
    rgw: 3 daemons active
 
  data:
    pools:   22 pools, 1816 pgs
    objects: 296k objects, 963 GB
    usage:   5222 GB used, 254 TB / 259 TB avail
    pgs:     0.220% pgs not active
             12408/606262 objects misplaced (2.047%)
             1763 active+clean
             29   active+remapped+backfill_wait
             14   active+remapped+backfilling
             5    active+clean+inconsistent
             3    peering
             1    active+recovery_wait
             1    activating+remapped
 
  io:
    client:   59450 kB/s rd, 4419 MB/s wr, 1095 op/s rd, 2848 op/s wr
    recovery: 253 MB/s, 210 keys/s, 123 objects/s
 

2018-07-05 14:21:07.867104 mon.dell1 [WRN] Health check failed: Degraded data redundancy: 7/605732 objects degraded (0.001%), 1 pg degraded (PG_DEGRADED)
2018-07-05 14:21:12.252395 mon.dell1 [INF] Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 7/605732 objects degraded (0.001%), 1 pg degraded)
2018-07-05 14:21:13.510741 mon.dell1 [WRN] Health check update: 12269/606262 objects misplaced (2.024%) (OBJECT_MISPLACED)
2018-07-05 14:21:13.510797 mon.dell1 [INF] Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 2 pgs peering)
2018-07-05 14:21:19.488864 mon.dell1 [WRN] Health check update: 11553/606262 objects misplaced (1.906%) (OBJECT_MISPLACED)
2018-07-05 14:21:25.502619 mon.dell1 [WRN] Health check update: 10504/606262 objects misplaced (1.733%) (OBJECT_MISPLACED)
2018-07-05 14:21:31.745600 mon.dell1 [WRN] Health check update: 10091/606262 objects misplaced (1.664%) (OBJECT_MISPLACED)
2018-07-05 14:21:36.779666 mon.dell1 [WRN] Health check update: 9309/606262 objects misplaced (1.535%) (OBJECT_MISPLACED)
2018-07-05 14:21:41.779947 mon.dell1 [WRN] Health check update: 8580/606262 objects misplaced (1.415%) (OBJECT_MISPLACED)
2018-07-05 14:21:46.816584 mon.dell1 [WRN] Health check update: 8215/606262 objects misplaced (1.355%) (OBJECT_MISPLACED)
2018-07-05 14:21:51.817014 mon.dell1 [WRN] Health check update: 7331/606262 objects misplaced (1.209%) (OBJECT_MISPLACED)
2018-07-05 14:21:56.817406 mon.dell1 [WRN] Health check update: 6929/606262 objects misplaced (1.143%) (OBJECT_MISPLACED)
2018-07-05 14:22:01.817820 mon.dell1 [WRN] Health check update: 6426/606262 objects misplaced (1.060%) (OBJECT_MISPLACED)
2018-07-05 14:22:06.818188 mon.dell1 [WRN] Health check update: 5787/606262 objects misplaced (0.955%) (OBJECT_MISPLACED)
2018-07-05 14:22:11.818606 mon.dell1 [WRN] Health check update: 5429/606262 objects misplaced (0.895%) (OBJECT_MISPLACED)
2018-07-05 14:22:16.818981 mon.dell1 [WRN] Health check update: 5165/606262 objects misplaced (0.852%) (OBJECT_MISPLACED)
2018-07-05 14:22:20.303513 osd.35 [ERR] 13.2ad missing primary copy of 13:b56abc11:::d9593962-fa39-406f-bc35-7e4fcac1be9f.44307.2__shadow_121_1530008810116503747%2fplatform-cms.rar.2~vf7AEOlGNYM1ggI6IhV-iu22oDDXcvS.5_1:head, will try copies on 0
2018-07-05 14:22:21.819353 mon.dell1 [WRN] Health check update: 4866/606262 objects misplaced (0.803%) (OBJECT_MISPLACED)
2018-07-05 14:22:26.819657 mon.dell1 [WRN] Health check update: 4586/606262 objects misplaced (0.756%) (OBJECT_MISPLACED)
2018-07-05 14:22:31.819983 mon.dell1 [WRN] Health check update: 4323/606262 objects misplaced (0.713%) (OBJECT_MISPLACED)
2018-07-05 14:22:36.820335 mon.dell1 [WRN] Health check update: 4113/606262 objects misplaced (0.678%) (OBJECT_MISPLACED)
2018-07-05 14:22:41.820676 mon.dell1 [WRN] Health check update: 3949/606262 objects misplaced (0.651%) (OBJECT_MISPLACED)
2018-07-05 14:22:46.821040 mon.dell1 [WRN] Health check update: 3788/606262 objects misplaced (0.625%) (OBJECT_MISPLACED)
2018-07-05 14:22:51.821395 mon.dell1 [WRN] Health check update: 3665/606262 objects misplaced (0.605%) (OBJECT_MISPLACED)
2018-07-05 14:22:56.821692 mon.dell1 [WRN] Health check update: 3440/606262 objects misplaced (0.567%) (OBJECT_MISPLACED)
2018-07-05 14:23:01.821999 mon.dell1 [WRN] Health check update: 3170/606266 objects misplaced (0.523%) (OBJECT_MISPLACED)
2018-07-05 14:23:06.822355 mon.dell1 [WRN] Health check update: 2956/606266 objects misplaced (0.488%) (OBJECT_MISPLACED)
2018-07-05 14:23:11.822752 mon.dell1 [WRN] Health check update: 2747/606270 objects misplaced (0.453%) (OBJECT_MISPLACED)
2018-07-05 14:23:16.823168 mon.dell1 [WRN] Health check update: 2615/606270 objects misplaced (0.431%) (OBJECT_MISPLACED)
2018-07-05 14:23:21.823523 mon.dell1 [WRN] Health check update: 2512/606270 objects misplaced (0.414%) (OBJECT_MISPLACED)
2018-07-05 14:23:26.823878 mon.dell1 [WRN] Health check update: 2409/606270 objects misplaced (0.397%) (OBJECT_MISPLACED)
2018-07-05 14:23:31.824214 mon.dell1 [WRN] Health check update: 2299/606270 objects misplaced (0.379%) (OBJECT_MISPLACED)
2018-07-05 14:23:36.824596 mon.dell1 [WRN] Health check update: 2194/606270 objects misplaced (0.362%) (OBJECT_MISPLACED)
2018-07-05 14:23:41.825037 mon.dell1 [WRN] Health check update: 2101/606270 objects misplaced (0.347%) (OBJECT_MISPLACED)
2018-07-05 14:23:46.825390 mon.dell1 [WRN] Health check update: 1939/606270 objects misplaced (0.320%) (OBJECT_MISPLACED)
2018-07-05 14:23:51.825725 mon.dell1 [WRN] Health check update: 1777/606270 objects misplaced (0.293%) (OBJECT_MISPLACED)
2018-07-05 14:23:56.826087 mon.dell1 [WRN] Health check update: 1612/606270 objects misplaced (0.266%) (OBJECT_MISPLACED)
2018-07-05 14:24:01.826439 mon.dell1 [WRN] Health check update: 1444/606270 objects misplaced (0.238%) (OBJECT_MISPLACED)
2018-07-05 14:24:06.826755 mon.dell1 [WRN] Health check update: 1315/606270 objects misplaced (0.217%) (OBJECT_MISPLACED)
2018-07-05 14:24:11.828343 mon.dell1 [WRN] Health check update: 1264/606270 objects misplaced (0.208%) (OBJECT_MISPLACED)
2018-07-05 14:24:16.828638 mon.dell1 [WRN] Health check update: 1214/606270 objects misplaced (0.200%) (OBJECT_MISPLACED)
2018-07-05 14:24:21.886644 mon.dell1 [WRN] Health check update: 1161/606270 objects misplaced (0.191%) (OBJECT_MISPLACED)
2018-07-05 14:24:26.887027 mon.dell1 [WRN] Health check update: 1110/606270 objects misplaced (0.183%) (OBJECT_MISPLACED)
2018-07-05 14:24:32.287725 mon.dell1 [WRN] Health check update: 1069/606270 objects misplaced (0.176%) (OBJECT_MISPLACED)
2018-07-05 14:24:39.839578 mon.dell1 [WRN] Health check update: 960/606270 objects misplaced (0.158%) (OBJECT_MISPLACED)
2018-07-05 14:24:45.851276 mon.dell1 [WRN] Health check update: 905/606272 objects misplaced (0.149%) (OBJECT_MISPLACED)
2018-07-05 14:24:51.911053 mon.dell1 [WRN] Health check update: 849/606272 objects misplaced (0.140%) (OBJECT_MISPLACED)
2018-07-05 14:24:57.960803 mon.dell1 [WRN] Health check update: 784/606272 objects misplaced (0.129%) (OBJECT_MISPLACED)
2018-07-05 14:25:05.887641 mon.dell1 [WRN] Health check update: 688/606272 objects misplaced (0.113%) (OBJECT_MISPLACED)
2018-07-05 14:25:11.945922 mon.dell1 [WRN] Health check update: 631/606272 objects misplaced (0.104%) (OBJECT_MISPLACED)
2018-07-05 14:25:16.946267 mon.dell1 [WRN] Health check update: 570/606272 objects misplaced (0.094%) (OBJECT_MISPLACED)
2018-07-05 14:25:21.993994 mon.dell1 [WRN] Health check update: 528/606272 objects misplaced (0.087%) (OBJECT_MISPLACED)
2018-07-05 14:25:26.994417 mon.dell1 [WRN] Health check update: 468/606272 objects misplaced (0.077%) (OBJECT_MISPLACED)
2018-07-05 14:25:31.994789 mon.dell1 [WRN] Health check update: 411/606272 objects misplaced (0.068%) (OBJECT_MISPLACED)
2018-07-05 14:25:36.995192 mon.dell1 [WRN] Health check update: 353/606272 objects misplaced (0.058%) (OBJECT_MISPLACED)
2018-07-05 14:25:42.009567 mon.dell1 [WRN] Health check update: 293/606272 objects misplaced (0.048%) (OBJECT_MISPLACED)
2018-07-05 14:25:47.009879 mon.dell1 [WRN] Health check update: 241/606272 objects misplaced (0.040%) (OBJECT_MISPLACED)
2018-07-05 14:25:52.010822 mon.dell1 [WRN] Health check update: 187/606272 objects misplaced (0.031%) (OBJECT_MISPLACED)
2018-07-05 14:25:57.011182 mon.dell1 [WRN] Health check update: 133/606272 objects misplaced (0.022%) (OBJECT_MISPLACED)
2018-07-05 14:26:02.035637 mon.dell1 [WRN] Health check update: 78/606272 objects misplaced (0.013%) (OBJECT_MISPLACED)
2018-07-05 14:26:07.035965 mon.dell1 [WRN] Health check update: 22/606272 objects misplaced (0.004%) (OBJECT_MISPLACED)
2018-07-05 14:26:12.011546 mon.dell1 [INF] Health check cleared: OBJECT_MISPLACED (was: 22/606272 objects misplaced (0.004%))

2.2 PG 修复

但不是数据迁移结束后就万事大吉了,可以通过下面这个命令看到,数据迁移后,有五个 pg 状态不正常,需要修复。

$ ceph health detail
HEALTH_ERR 6 scrub errors; Possible data damage: 5 pgs inconsistent
OSD_SCRUB_ERRORS 6 scrub errors
PG_DAMAGED Possible data damage: 5 pgs inconsistent
    pg 13.cd is active+clean+inconsistent, acting [20,35]
    pg 13.244 is active+clean+inconsistent, acting [35,22]
    pg 13.270 is active+clean+inconsistent, acting [35,14]
    pg 13.308 is active+clean+inconsistent, acting [35,17]
    pg 13.34f is active+clean+inconsistent, acting [11,35]

执行 repair 命令来修复,如果还是不成功,可以使用 scrub 来进行数据清理。

$ ceph pg repair 13.cd
$ ceph pg scrub 13.cd

2.3 关闭 OSD 进程

数据迁移至此算是完成了,但 osd 进程还是跑着的。

 0   hdd   5.52620         osd.0      up        0 1.00000

接下来需要登录到 OSD 服务器上关闭掉该进程。

$ ssh osd8
$ sudo systemctl stop ceph-osd@0

现在 osd 进程的状态已经已经是 down 了。

 0   hdd   5.52620         osd.0    down        0 1.00000 

2.4 删除 OSD

最后执行 purge 命令,将该 osd 从 CRUSH map 中彻底删掉,至此,单个 OSD 的删除终于完成了。

$ ceph osd purge 0 --yes-i-really-mean-it
purged osd.0

对了,最后,如果 /etc/ceph/ceph.conf 中由对应的该 osd 的信息,记得要一起删除。

3. 参考文档

本文参与腾讯云自媒体分享计划,欢迎正在阅读的你也加入,一起分享。

发表于

我来说两句

0 条评论
登录 后参与评论

相关文章

来自专栏圣杰的专栏

.NET Core+MySql+Nginx 容器化部署

1. 引言 上两节我们通过简单的demo学习了docker的基本操作。这一节我们来一个进阶学习,完成ASP.NET Core + MySql + Nginx的容...

4348
来自专栏杨建荣的学习笔记

使用shell定制awr脚本(r3笔记第32天)

大家在做性能问题诊断的时候,awr是不可或缺的工具,使用?/rdbms/admin/awrrpt.sql可能大家使用的多了,可能有时候感觉输入参数还是有些太繁琐...

2844
来自专栏Golang语言社区

使用Docker和热加载运行Go API

This is a quick discussion of how to set up a local development environment for ...

1241
来自专栏程序员同行者

Docker部署Vue 工程包

这样前端工程镜像就build好了,可以执行docker run -d -p9528:9528 dist:v0.1启动

1282
来自专栏运维

linux下连接windows2003 ppoe 服务器

1,在CentOS6.3下用 /usr/sbin/pppd  pty "/usr/sbin/pptp publicip --nolaunchpppd" f...

1292
来自专栏木制robot技术杂谈

Ubuntu 使用 Docker 安装 Gitlab

最近帮公司重新搭建了 Gitlab,中间遇到了一些坑,折腾了不少时间,在此记录供大家参考。

3234
来自专栏乐沙弥的世界

Oracle 11g RAC CRS-4535/ORA-15077

    新安装了Oracle 11g rac之后,不知道是什么原因导致第二个节点上的crsd无法启动?其错误消息是CRS-4535: Cannot commun...

903
来自专栏王亚昌的专栏

How to build your own ubuntu image with docker?

docker run -d -p 222:22 ubuntu-sshd-admin

1112
来自专栏乐沙弥的世界

Oracle 11g RAC crs_stat 命令结果完整显示

Oracle 11g RAC中crs_stat命令较之前的版本多出了很多新的不同的资源类型,缺省情况下,使用crs_stat -t来查看资源是密密麻麻一大片,看...

2411
来自专栏乐沙弥的世界

CRS-1006 , CRS-0215 故障一例

    安装好sles 10 sp3 + Oracle 10g RAC之后,在配置监听器时,总是提示主机bo2dbp上的监听服务已经在运行,忽略错误之后手动在b...

613

扫码关注云+社区

领取腾讯云代金券