前往小程序,Get更优阅读体验!
立即前往
首页
学习
活动
专区
工具
TVP
发布
社区首页 >专栏 >Harbor-hlem镜像库重新部署后PV数据恢复

Harbor-hlem镜像库重新部署后PV数据恢复

作者头像
三杯水Plus
发布2020-07-16 15:44:19
1.2K0
发布2020-07-16 15:44:19
举报
文章被收录于专栏:运维运维

起因

开发反馈habor镜像库登陆不了,初步查看是证书过期了。

解决方案 之前Harbor-helm部署镜像库文档可以回顾链接https://minminmsn.com/middleware/698/ 1.首先新建新证书的secret

代码语言:javascript
复制
[root@elasticsearch01 harbor-helm]#  kubectl  create secret tls ingress-secret2021 --key minminmsnauto.key --cert minminmsnauto.crt 

2.然后修改harbor-helm的value.yaml,把secretName替换下

代码语言:javascript
复制
[root@elasticsearch01 harbor-helm]# head -n 20 values.yaml
expose:
  # Set the way how to expose the service. Set the type as "ingress", 
  # "clusterIP" or "nodePort" and fill the information in the corresponding 
  # section
  type: ingress
  tls:
    # Enable the tls or not. Note: if the type is "ingress" and the tls 
    # is disabled, the port must be included in the command when pull/push
    # images. Refer to https://github.com/goharbor/harbor/issues/5291 
    # for the detail.
    enabled: true
    # Fill the name of secret if you want to use your own TLS certificate
    # and private key. The secret must contain keys named tls.crt and 
    # tls.key that contain the certificate and private key to use for TLS
    # The certificate and private key will be generated automatically if 
    # it is not set
    secretName: "ingress-secret2021"
    # By default, the Notary service will use the same cert and key as
    # described above. Fill the name of secret if you want to use a 
    # separated one. Only needed when the type is "ingress".

3.最后使用helm upgrade更新版本

代码语言:javascript
复制
[root@elasticsearch01 harbor-helm]#  helm upgrade  minminmsn . -f values.yaml

到这个时候应该能解决需求,可是事与愿违,不知道哪儿除了问题,这时登陆Harbor证书问题是解决了,但是项目及库访问不了提示内部错误,看Pod的运行状态也都是Running。 最后打算使用helm先delete掉再install,但是这样创建的harbor看起来一切正常,实际上是个初始化环境,是自动生成的新PV并没有原来的数据。此时发现原来的PV还在,下面就开始找PV恢复的方案。

调整PV状态

1.查询此时PV与PVC状态

代码语言:javascript
复制
[root@elasticsearch01 harbor-helm]# kubectl get pv
NAME                                       CAPACITY   ACCESS MODES   RECLAIM POLICY   STATUS     CLAIM                                            STORAGECLASS   REASON   AGE                    9h
pvc-e7967cfe-7ded-11e9-a09d-52540089b2b6   50Gi       RWO            Retain           Released   default/minminmsn-harbor-chartmuseum                rbd                     417d
pvc-e7974d1c-7ded-11e9-a09d-52540089b2b6   20Gi       RWO            Retain           Released   default/minminmsn-harbor-jobservice                 rbd                     417d
pvc-e7985b55-7ded-11e9-a09d-52540089b2b6   2000Gi     RWO            Retain           Released   default/minminmsn-harbor-registry                   rbd                     417d
pvc-e7d38097-7ded-11e9-a09d-52540089b2b6   20Gi       RWO            Retain           Released   default/database-data-minminmsn-harbor-database-0   rbd                     417d
pvc-e7da3f3c-7ded-11e9-a09d-52540089b2b6   20Gi       RWO            Retain           Released   default/data-minminmsn-harbor-redis-0               rbd                     417d
[root@elasticsearch01 harbor-helm]# kubectl get pvc
NAME                                     STATUS   VOLUME                                     CAPACITY   ACCESS MODES   STORAGECLASS   AGE
data-minminmsn-harbor-redis-0               Bound    pvc-6cd422e4-c5f0-11ea-9386-52540089b2b6   20Gi       RWO            rbd            9h
database-data-minminmsn-harbor-database-0   Bound    pvc-6ccda00b-c5f0-11ea-9386-52540089b2b6   20Gi       RWO            rbd            9h
minminmsn-harbor-chartmuseum                Bound    pvc-6c903857-c5f0-11ea-9386-52540089b2b6   50Gi       RWO            rbd            9h
minminmsn-harbor-jobservice                 Bound    pvc-6c91d1a4-c5f0-11ea-9386-52540089b2b6   20Gi       RWO            rbd            9h
minminmsn-harbor-registry                   Bound    pvc-6c92bfc0-c5f0-11ea-9386-52540089b2b6   500Gi      RWO            rbd            9h

2.修改PV状态 先把PV的状态由Released改变成 备注:默认创建的PV的回收策略是Delete就是用完就删除,之前特意把RECLAIM POLICY改为了Retain,在线修改PV回收策略可以参考文档https://minminmsn.com/cloud/1091/。否则这里Helm Delete后就会自动删除PV,就没有后来这篇PV数据恢复操作了。 在线编辑PV,需要把其中claimRef这段删除,这样状态就可以变成Available了。

代码语言:javascript
复制
  claimRef:
    apiVersion: v1
    kind: PersistentVolumeClaim
    name: minminmsn-harbor-chartmuseum
    namespace: default
    resourceVersion: "91736092"
    uid: b31ec8ca-c649-11ea-9386-52540089b2b6
  persistentVolumeReclaimPolicy: Retain

具体如下修改

代码语言:javascript
复制
[root@elasticsearch01 harbor-helm]# kubectl edit pv pvc-e7967cfe-7ded-11e9-a09d-52540089b2b6 
# Please edit the object below. Lines beginning with a '#' will be ignored,
# and an empty file will abort the edit. If an error occurs while saving this file will be
# reopened with the relevant failures.
#
apiVersion: v1
kind: PersistentVolume
metadata:
  annotations:
    pv.kubernetes.io/bound-by-controller: "yes"
    pv.kubernetes.io/provisioned-by: ceph.com/rbd
    rbdProvisionerIdentity: ceph.com/rbd
  creationTimestamp: "2019-05-24T06:33:55Z"
  finalizers:
  - kubernetes.io/pv-protection
  name: pvc-e7967cfe-7ded-11e9-a09d-52540089b2b6
  resourceVersion: "91736100"
  selfLink: /api/v1/persistentvolumes/pvc-e7967cfe-7ded-11e9-a09d-52540089b2b6
  uid: e7ade7f7-7ded-11e9-a09d-52540089b2b6
spec:
  accessModes:
  - ReadWriteOnce
  capacity:
    storage: 50Gi
  claimRef:
    apiVersion: v1
    kind: PersistentVolumeClaim
    name: minminmsn-harbor-chartmuseum
    namespace: default
    resourceVersion: "91736092"
    uid: b31ec8ca-c649-11ea-9386-52540089b2b6
  persistentVolumeReclaimPolicy: Retain
  rbd:
    image: kubernetes-dynamic-pvc-e79b34d3-7ded-11e9-ac1b-02420afe4905
    keyring: /etc/ceph/keyring
    monitors:
    - 10.0.4.8:6789
    pool: rbd-k8s
    secretRef:
      name: ceph-secret
      namespace: default
    user: admin
  storageClassName: rbd
  volumeMode: Filesystem
status:
  phase: Released

3.其他四个PV同样操作

代码语言:javascript
复制
[root@elasticsearch01 harbor-helm]# kubectl edit pv pvc-e7974d1c-7ded-11e9-a09d-52540089b2b6
[root@elasticsearch01 harbor-helm]# kubectl edit pv pvc-e7985b55-7ded-11e9-a09d-52540089b2b6
[root@elasticsearch01 harbor-helm]# kubectl edit pv pvc-e7d38097-7ded-11e9-a09d-52540089b2b6
[root@elasticsearch01 harbor-helm]# kubectl edit pv pvc-e7da3f3c-7ded-11e9-a09d-52540089b2b6

4.查看效果 现在看PV的STATUS已经变成了Available,然后CLAIM也变空了,这样就可以在后面绑定使用了

代码语言:javascript
复制
[root@elasticsearch01 harbor-helm]# kubectl get pv
NAME                                       CAPACITY   ACCESS MODES   RECLAIM POLICY   STATUS      CLAIM                       STORAGECLASS   REASON   AGE
pvc-e7967cfe-7ded-11e9-a09d-52540089b2b6   50Gi       RWO            Retain           Available                               rbd                     417d
pvc-e7974d1c-7ded-11e9-a09d-52540089b2b6   20Gi       RWO            Retain           Available                               rbd                     417d
pvc-e7985b55-7ded-11e9-a09d-52540089b2b6   2000Gi     RWO            Retain           Available                               rbd                     417d
pvc-e7d38097-7ded-11e9-a09d-52540089b2b6   20Gi       RWO            Retain           Available                               rbd                     417d
pvc-e7da3f3c-7ded-11e9-a09d-52540089b2b6   20Gi       RWO            Retain           Available                               rbd                     417d

创建PVC

1.先设置好PVC及PV对应关系

代码语言:javascript
复制
[root@elasticsearch01 yaml]# cat minminmsn.pvc 
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: minminmsn-harbor-registry
spec:
  accessModes:
    - ReadWriteOnce
  storageClassName: "rbd"
  resources:
    requests:
      storage: 2000Gi
  volumeName: "pvc-e7985b55-7ded-11e9-a09d-52540089b2b6"
---
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: minminmsn-harbor-jobservice
spec:
  accessModes:
    - ReadWriteOnce
  storageClassName: "rbd"
  resources:
    requests:
      storage: 20Gi
  volumeName: "pvc-e7974d1c-7ded-11e9-a09d-52540089b2b6"
---
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: minminmsn-harbor-chartmuseum
spec:
  accessModes:
    - ReadWriteOnce
  storageClassName: "rbd"
  resources:
    requests:
      storage: 50Gi
  volumeName: "pvc-e7967cfe-7ded-11e9-a09d-52540089b2b6"
---
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: database-data-minminmsn-harbor-database-0
spec:
  accessModes:
    - ReadWriteOnce
  storageClassName: "rbd"
  resources:
    requests:
      storage: 20Gi
  volumeName: "pvc-e7d38097-7ded-11e9-a09d-52540089b2b6"
---
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: data-minminmsn-harbor-redis-0
spec:
  accessModes:
    - ReadWriteOnce
  storageClassName: "rbd"
  resources:
    requests:
      storage: 20Gi
  volumeName: "pvc-e7da3f3c-7ded-11e9-a09d-52540089b2b6"

2.创建PVC

代码语言:javascript
复制
[root@elasticsearch01 yaml]# kubectl apply -f minminmsn.pvc 
persistentvolumeclaim/minminmsn-harbor-registry created
persistentvolumeclaim/minminmsn-harbor-jobservice created
persistentvolumeclaim/minminmsn-harbor-chartmuseum created
persistentvolumeclaim/database-data-minminmsn-harbor-database-0 created
persistentvolumeclaim/data-minminmsn-harbor-redis-0 created

3.检查PV与PVC

代码语言:javascript
复制
[root@elasticsearch01 yaml]# kubectl get pv
NAME                                       CAPACITY   ACCESS MODES   RECLAIM POLICY   STATUS   CLAIM                                            STORAGECLASS   REASON   AGE
pvc-e7967cfe-7ded-11e9-a09d-52540089b2b6   50Gi       RWO            Retain           Bound    default/minminmsn-harbor-chartmuseum                rbd                     417d
pvc-e7974d1c-7ded-11e9-a09d-52540089b2b6   20Gi       RWO            Retain           Bound    default/minminmsn-harbor-jobservice                 rbd                     417d
pvc-e7985b55-7ded-11e9-a09d-52540089b2b6   2000Gi     RWO            Retain           Bound    default/minminmsn-harbor-registry                   rbd                     417d
pvc-e7d38097-7ded-11e9-a09d-52540089b2b6   20Gi       RWO            Retain           Bound    default/database-data-minminmsn-harbor-database-0   rbd                     417d
pvc-e7da3f3c-7ded-11e9-a09d-52540089b2b6   20Gi       RWO            Retain           Bound    default/data-minminmsn-harbor-redis-0               rbd                     417d
[root@elasticsearch01 yaml]# kubectl get pvc
NAME                                     STATUS    VOLUME                                     CAPACITY   ACCESS MODES   STORAGECLASS   AGE
ceph-rbd-pv-claim                        Bound     ceph-rbd-pv                                20Gi       RWO                           540d
data-minminmsn-harbor-redis-0               Pending   pvc-e7da3f3c-7ded-11e9-a09d-52540089b2b6   0                         rbd            12s
database-data-minminmsn-harbor-database-0   Pending   pvc-e7d38097-7ded-11e9-a09d-52540089b2b6   0                         rbd            12s
minminmsn-harbor-chartmuseum                Pending   pvc-e7967cfe-7ded-11e9-a09d-52540089b2b6   0                         rbd            12s
minminmsn-harbor-jobservice                 Pending   pvc-e7974d1c-7ded-11e9-a09d-52540089b2b6   0                         rbd            12s
minminmsn-harbor-registry                   Bound     pvc-e7985b55-7ded-11e9-a09d-52540089b2b6   2000Gi     RWO            rbd            12s
[root@elasticsearch01 yaml]# kubectl describe pvc minminmsn-harbor-registry
Name:          minminmsn-harbor-registry
Namespace:     default
StorageClass:  rbd
Status:        Bound
Volume:        pvc-e7985b55-7ded-11e9-a09d-52540089b2b6
Labels:        <none>
Annotations:   kubectl.kubernetes.io/last-applied-configuration:
                 {"apiVersion":"v1","kind":"PersistentVolumeClaim","metadata":{"annotations":{},"name":"minminmsn-harbor-registry","namespace":"default"},"spe...
               pv.kubernetes.io/bind-completed: yes
Finalizers:    [kubernetes.io/pvc-protection]
Capacity:      2000Gi
Access Modes:  RWO
VolumeMode:    Filesystem
Events:        <none>
Mounted By:    <none>

使用Hlem重新部署Harbor镜像库

1.部署前先删除版本

代码语言:javascript
复制
[root@elasticsearch01 harbor-helm]# helm delete --purge minminmsn
helm delete --purge minminmsn
release "minminmsn" deleted

2.修改Harbor-helm的values.yaml中PVC相关值 注意existingClaim: ""由空值改成上面生成的PVC名字,注意对应关系,其他不变,具体变更如下

代码语言:javascript
复制
persistence:
  enabled: true
  # Setting it to "keep" to avoid removing PVCs during a helm delete 
  # operation. Leaving it empty will delete PVCs after the chart deleted
  resourcePolicy: "keep"
  persistentVolumeClaim:
    registry:
      # Use the existing PVC which must be created manually before bound
      existingClaim: "minminmsn-harbor-registry"
      # Specify the "storageClass" used to provision the volume. Or the default
      # StorageClass will be used(the default).
      # Set it to "-" to disable dynamic provisioning
      storageClass: "rbd"
      subPath: ""
      accessMode: ReadWriteOnce
      size: 2000Gi
    chartmuseum:
      existingClaim: "minminmsn-harbor-chartmuseum"
      storageClass: "rbd"
      subPath: ""
      accessMode: ReadWriteOnce
      size: 50Gi
    jobservice:
      existingClaim: "minminmsn-harbor-jobservice"
      storageClass: "rbd"
      subPath: ""
      accessMode: ReadWriteOnce
      size: 20Gi
    # If external database is used, the following settings for database will 
    # be ignored
    database:
      existingClaim: "database-data-minminmsn-harbor-database-0"
      storageClass: "rbd"
      subPath: ""
      accessMode: ReadWriteOnce
      size: 20Gi
    # If external Redis is used, the following settings for Redis will 
    # be ignored
    redis:
      existingClaim: "data-minminmsn-harbor-redis-0"
      storageClass: "rbd"
      subPath: ""
      accessMode: ReadWriteOnce
      size: 20Gi

3.重新部署

代码语言:javascript
复制
[root@elasticsearch01 harbor-helm]# helm  install . --name minminmsn
NAME:   minminmsn
LAST DEPLOYED: Wed Jul 15 11:18:13 2020
NAMESPACE: default
STATUS: DEPLOYED

RESOURCES:
==> v1/Service
NAME                         TYPE       CLUSTER-IP      EXTERNAL-IP  PORT(S)            AGE
minminmsn-harbor-adminserver    ClusterIP  10.254.58.23    <none>       80/TCP             1s
minminmsn-harbor-chartmuseum    ClusterIP  10.254.154.44   <none>       80/TCP             1s
minminmsn-harbor-clair          ClusterIP  10.254.25.107   <none>       6060/TCP           1s
minminmsn-harbor-core           ClusterIP  10.254.56.153   <none>       80/TCP             1s
minminmsn-harbor-database       ClusterIP  10.254.65.18    <none>       5432/TCP           1s
minminmsn-harbor-jobservice     ClusterIP  10.254.81.97    <none>       80/TCP             1s
minminmsn-harbor-notary-server  ClusterIP  10.254.99.90    <none>       4443/TCP           1s
minminmsn-harbor-notary-signer  ClusterIP  10.254.175.105  <none>       7899/TCP           1s
minminmsn-harbor-portal         ClusterIP  10.254.242.113  <none>       80/TCP             1s
minminmsn-harbor-redis          ClusterIP  10.254.127.40   <none>       6379/TCP           1s
minminmsn-harbor-registry       ClusterIP  10.254.158.222  <none>       5000/TCP,8080/TCP  1s

==> v1/Deployment
NAME                         DESIRED  CURRENT  UP-TO-DATE  AVAILABLE  AGE
minminmsn-harbor-adminserver    1        1        1           0          1s
minminmsn-harbor-chartmuseum    1        1        1           0          1s
minminmsn-harbor-clair          1        0        0           0          1s
minminmsn-harbor-core           1        0        0           0          1s
minminmsn-harbor-jobservice     1        0        0           0          1s
minminmsn-harbor-notary-server  1        0        0           0          1s
minminmsn-harbor-notary-signer  1        0        0           0          1s
minminmsn-harbor-portal         1        0        0           0          1s
minminmsn-harbor-registry       1        0        0           0          1s

==> v1/StatefulSet
NAME                    DESIRED  CURRENT  AGE
minminmsn-harbor-database  1        1        1s
minminmsn-harbor-redis     1        1        1s

==> v1beta1/Ingress
NAME                   HOSTS                                                    ADDRESS  PORTS  AGE
minminmsn-harbor-ingress  core-harbor.minminmsn.com,notary-harbor.minminmsn.com  80, 443  1s

==> v1/Pod(related)
NAME                                          READY  STATUS             RESTARTS  AGE
minminmsn-harbor-adminserver-b5d58db8c-wmrbd     0/1    ContainerCreating  0         1s
minminmsn-harbor-chartmuseum-7c6b9d4977-94rhb    0/1    Pending            0         1s
minminmsn-harbor-clair-54465ff7dd-d7bxx          0/1    Pending            0         1s
minminmsn-harbor-core-587cc5d9b5-2xxl9           0/1    Pending            0         1s
minminmsn-harbor-jobservice-764bb697d-wsxqx      0/1    Pending            0         1s
minminmsn-harbor-notary-server-77fbb84fcc-2bw7c  0/1    Pending            0         1s
minminmsn-harbor-notary-signer-8466d68f5b-klv76  0/1    Pending            0         1s
minminmsn-harbor-database-0                      0/1    Pending            0         1s
minminmsn-harbor-redis-0                         0/1    Pending            0         1s

==> v1/Secret
NAME                       TYPE    DATA  AGE
minminmsn-harbor-adminserver  Opaque  4     1s
minminmsn-harbor-chartmuseum  Opaque  1     1s
minminmsn-harbor-core         Opaque  4     1s
minminmsn-harbor-database     Opaque  1     1s
minminmsn-harbor-jobservice   Opaque  1     1s
minminmsn-harbor-registry     Opaque  1     1s

==> v1/ConfigMap
NAME                         DATA  AGE
minminmsn-harbor-adminserver    39    1s
minminmsn-harbor-chartmuseum    24    1s
minminmsn-harbor-clair          1     1s
minminmsn-harbor-core           1     1s
minminmsn-harbor-jobservice     1     1s
minminmsn-harbor-notary-server  5     1s
minminmsn-harbor-registry       2     1s

NOTES:
Please wait for several minutes for Harbor deployment to complete.
Then you should be able to visit the Harbor portal at https://core-harbor.minminmsn.com. 
For more details, please visit https://github.com/goharbor/harbor.

3.查看新生成Pods的信息
[root@elasticsearch01 harbor-helm]# kubectl get pods
NAME                                           READY   STATUS              RESTARTS   AGE
minminmsn-harbor-adminserver-b5d58db8c-wmrbd      0/1     ContainerCreating   0          9s
minminmsn-harbor-chartmuseum-7c6b9d4977-94rhb     0/1     ContainerCreating   0          9s
minminmsn-harbor-clair-54465ff7dd-d7bxx           0/1     Running             0          9s
minminmsn-harbor-core-587cc5d9b5-2xxl9            0/1     Running             0          9s
minminmsn-harbor-database-0                       0/1     Init:0/1            0          9s
minminmsn-harbor-jobservice-764bb697d-wsxqx       0/1     ContainerCreating   0          9s
minminmsn-harbor-notary-server-77fbb84fcc-2bw7c   0/1     ContainerCreating   0          9s
minminmsn-harbor-notary-signer-8466d68f5b-klv76   0/1     ContainerCreating   0          9s
minminmsn-harbor-portal-64cf8b9b69-xm8nl          0/1     ContainerCreating   0          8s
minminmsn-harbor-redis-0                          0/1     ContainerCreating   0          9s
minminmsn-harbor-registry-755746c5bb-q8m55        0/2     ContainerCreating   0          8s

再等2分钟查看就上恢复了

代码语言:javascript
复制
[root@elasticsearch01 harbor-helm]# kubectl get pods
NAME                                           READY   STATUS    RESTARTS   AGE
jenkins-0                                      1/1     Running   0          62d
rbd-provisioner-67b4857bcd-rjwlg               1/1     Running   0          61d
minminmsn-harbor-adminserver-b5d58db8c-wmrbd      1/1     Running   1          2m33s
minminmsn-harbor-chartmuseum-7c6b9d4977-94rhb     1/1     Running   0          2m33s
minminmsn-harbor-clair-54465ff7dd-d7bxx           1/1     Running   1          2m33s
minminmsn-harbor-core-587cc5d9b5-2xxl9            1/1     Running   1          2m33s
minminmsn-harbor-database-0                       1/1     Running   0          2m33s
minminmsn-harbor-jobservice-764bb697d-wsxqx       1/1     Running   0          2m33s
minminmsn-harbor-notary-server-77fbb84fcc-2bw7c   1/1     Running   0          2m33s
minminmsn-harbor-notary-signer-8466d68f5b-klv76   1/1     Running   0          2m33s
minminmsn-harbor-portal-64cf8b9b69-xm8nl          1/1     Running   0          2m32s
minminmsn-harbor-redis-0                          1/1     Running   0          2m33s
minminmsn-harbor-registry-755746c5bb-q8m55        2/2     Running   0          2m32s

4.Harbor控制验证 证书更新了项目也恢复了 https://core-harbor.minminmsn.com/harbor/projects

Harbor-hlem镜像库重新部署后PV数据恢复
Harbor-hlem镜像库重新部署后PV数据恢复
本文参与 腾讯云自媒体分享计划,分享自作者个人站点/博客。
原始发表:2020-07-15 ,如有侵权请联系 cloudcommunity@tencent.com 删除

本文分享自 作者个人站点/博客 前往查看

如有侵权,请联系 cloudcommunity@tencent.com 删除。

本文参与 腾讯云自媒体分享计划  ,欢迎热爱写作的你一起参与!

评论
登录后参与评论
0 条评论
热度
最新
推荐阅读
相关产品与服务
云数据库 Redis
腾讯云数据库 Redis(TencentDB for Redis)是腾讯云打造的兼容 Redis 协议的缓存和存储服务。丰富的数据结构能帮助您完成不同类型的业务场景开发。支持主从热备,提供自动容灾切换、数据备份、故障迁移、实例监控、在线扩容、数据回档等全套的数据库服务。
领券
问题归档专栏文章快讯文章归档关键词归档开发者手册归档开发者手册 Section 归档