[pve][ceph]pve+ceph节点重装

注意：
逐一重装系统／重新加入PVE集群／重新加入CEPH集群；
先禁用ceph的recover，避免其因OSD丢失重建：任一节点执行：
for i in noout nobackfill norecover;do ceph osd set $i;done
重新加入pve集群前确保已经从集群中删除

systemctl stop pvestatd.service
systemctl stop pvedaemon.service
systemctl stop pve-cluster.service

tar -czf /root/bak/pve-cluster-backup.tar.gz /var/lib/pve-cluster
tar -czf /root/bak/ssh-backup.tar.gz /root/.ssh
tar -czf /root/bak/corosync-backup.tar.gz /etc/corosync
cp /etc/hosts /root/bak/
cp /etc/network/interfaces /root/bak/

tar cjvf bak-$(hostname).tgz /root/bak

pvecm nodes
pvecm delnode xxx
pvecm updatecerts

tar xjvf bak-xxx.tgz
cp /root/bak/hosts /etc/hosts
cp /root/bak/interfaces /etc/network/interfaces

systemctl stop pvestatd.service
systemctl stop pvedaemon.service
systemctl stop pve-cluster.service

cd / ; tar -xvzf /root/bak/ssh-backup.tar.gz

rm -rf /var/lib/pve-cluster
cd / ; tar -xvzf /root/bak/pve-cluster-backup.tar.gz

rm -rf /etc/corosync
cd / ; tar -xvzf /root/bak/corosync-backup.tar.gz

pvecm add xxx.xxx.xxx.xxx -force
pvecm updatecerts

ceph mon stat
ceph mon rm xxx

ceph-volume lvm activate --all

pvecm expected 1
pvecm delete <dead node>
pvecm updatecerts

Permission denied (publickey).
TASK ERROR: Failed to run vncproxy.

for i in noout nobackfill norecover;do ceph osd unset $i;done