TKE中使用helm搭建Rancher并搭建prometheus+grafana

原创

马凌鑫

发布于 2019-05-08 17:10:50

4.1K1

发布于 2019-05-08 17:10:50

文章被收录于专栏：云知识学习云知识学习

上一篇写了如何搭建helm ，TKE搭建helm搭建

由于目前TKE已经集成了helm，用户只需在控制台点击安装便会下发tiller、swift

已省去了搭建的过程，非常方便，但是目前共有的chart不一定能满足所有人的需求，所以本篇以拓展应用为主，讲述如何在TKE集群中通过集成的helm搭建应用

接下来进入主题，如何用helm搭建rancher

Rancher是什么Rancher是一个开源的企业级容器管理平台。通过Rancher,企业再也不必自己使用一系列的开源软件去从头搭建容器服务平台

官方文档：https://www.cnrancher.com/docs/rancher/v2.x/cn/installation/server-tags/

一、下载helm客户端至节点

由于控制台的安装只下发了tiller、swift，还需手动下载helm client

$ curl -O https://storage.googleapis.com/kubernetes-helm/helm-v2.10.0-linux-amd64.tar.gz
$ tar xzvf helm-v2.10.0-linux-amd64.tar.gz
$ sudo cp linux-amd64/helm /usr/local/bin/helm

注意，这里默认的是v2.10.0版本，client与server的版本务必对齐不然会失败。

$ helm version
Client: &version.Version{SemVer:"v2.10.0", GitCommit:"9ad53aac42165a5fadc6c87be0dea6b115f93090", GitTreeState:"clean"}
Server: &version.Version{SemVer:"v2.10.0", GitCommit:"9ad53aac42165a5fadc6c87be0dea6b115f93090", GitTreeState:"clean"}

二、初始化helm

当tiller已存在的时候需加上--client-only

You might need to run helm init (or helm init --client-only if tiller is already installed)

$ helm init --client-only
Creating /root/.helm/repository/repositories.yaml 
Adding stable repo with URL: https://kubernetes-charts.storage.googleapis.com 
Adding local repo with URL: http://127.0.0.1:8879/charts 
$HELM_HOME has been configured at /root/.helm.
Not installing Tiller due to 'client-only' flag having been set
Happy Helming!

三、添加rancher仓库

类型	添加仓库命令	仓库描述
rancher-latest	helm repo add rancher-latest https://releases.rancher.com/server-charts/latest	Rancher server最新版Helm charts仓库，建议此仓库版本用于测试环境。
rancher-stable	helm repo add rancher-stable https://releases.rancher.com/server-charts/stable	Rancher server稳定版Helm charts仓库，此仓库版本推荐用于生产环境。

添加stable版本

$ helm repo add rancher-stable https://releases.rancher.com/server-charts/stable
"rancher-stable" has been added to your repositories

查看repo

$ helm repo list
NAME            URL                                              
stable          https://kubernetes-charts.storage.googleapis.com 
local           http://127.0.0.1:8879/charts                     
rancher-stable  https://releases.rancher.com/server-charts/stable

四、安装cert-manager

helm install stable/cert-manager \
--name cert-manager \
--namespace kube-system \
--version v0.5.2

五、安装rancher-stable

helm install rancher-stable/rancher  \
--name rancher \
--namespace cattle-system \
--set hostname=rancher.my.org

------------------------------------------------

$ helm install rancher-stable/rancher   --name rancher   --namespace cattle-system   --set hostname=rancher.my.org
NAME:   rancher
LAST DEPLOYED: Wed May  8 12:13:27 2019
NAMESPACE: cattle-system
STATUS: DEPLOYED

RESOURCES:
==> v1/Service
NAME     TYPE       CLUSTER-IP    EXTERNAL-IP  PORT(S)  AGE
rancher  ClusterIP  10.3.255.165  <none>       80/TCP   0s

==> v1/Deployment
NAME     DESIRED  CURRENT  UP-TO-DATE  AVAILABLE  AGE
rancher  3        0        0           0          0s

==> v1beta1/Ingress
NAME     HOSTS           ADDRESS  PORTS  AGE
rancher  rancher.my.org  80, 443  0s

==> v1alpha1/Issuer
NAME     AGE
rancher  0s

==> v1/Pod(related)
NAME                      READY  STATUS             RESTARTS  AGE
rancher-7bc85fd5ff-98mv6  0/1    ContainerCreating  0         0s
rancher-7bc85fd5ff-m87m2  0/1    ContainerCreating  0         0s
rancher-7bc85fd5ff-s54fx  0/1    ContainerCreating  0         0s

==> v1/ServiceAccount
NAME     SECRETS  AGE
rancher  1        0s

==> v1/ClusterRoleBinding
NAME     AGE
rancher  0s


NOTES:
Rancher Server has been installed.

NOTE: Rancher may take several minutes to fully initialize. Please standby while Certificates are being issued and Ingress comes up.

Check out our docs at https://rancher.com/docs/rancher/v2.x/en/

Browse to https://rancher.my.org

Happy Containering!

输入以上指令后会有详细信息过程的反馈，从反馈中可以看到rancher的chart创建了Service、Deployment、Ingress、Issuer、ServiceAccount、ClusterRoleBinding

通过查看pod可以发现 rancher已经以pod形式启动

$kubectl -n cattle-system rollout status deploy/rancher
Waiting for deployment "rancher" rollout to finish: 0 of 3 updated replicas are available...
deployment "rancher" successfully rolled out

$ kubectl get pod -n cattle-system
NAME                       READY     STATUS    RESTARTS   AGE
rancher-7bc85fd5ff-98mv6   1/1       Running   1          4m
rancher-7bc85fd5ff-m87m2   1/1       Running   1          4m
rancher-7bc85fd5ff-s54fx   1/1       Running   1          4m

创建完毕后需要做一些修改，符合TKE特性才能正常访问

六、修改配置

a. 修改rancher service type

rancher chart包创建的service type类型默认是clusterIP，需要修改为nodeport形式，否则ingress将无法转发请求至service

选择更新访问方式，选择主机端口访问，主机端口若不填写将从30000-32767中随机挑选一个。

b. 修改ingress

来到ingress的界面中可以看到ingress的 vip 一直在创建中

通过查看事件发现有warning事件：Error during sync: Secret tls-rancher-ingress has no qcloud cert id

说明这个证书没找到，需要将rancher的证书添加至腾讯云中

c. 添加证书至腾讯云中

在rancher v2.x版本后默认是配置了ssl证书的，在第四步中创建的cert-manager就是给rancher颁发证书

Rancher Server is designed to be secure by default and requires SSL/TLS configuration.

获取证书

$ kubectl get secret tls-rancher-ingress -n cattle-system -o yaml
apiVersion: v1
data:
  tls.crt: LS0tLS1CRUdJTiBDRVJUSUZJQ0FURS0tLS0tCk1JSURIakNDQWdhZ0F3SUJBZ0lSQU8rQUhOcFgvR0FPbEt1LzdBVmtWeTB3RFFZSktvWklodmNOQVFFTEJRQXcKS0RFU01CQUdBMVVFQ2hNSmRHaGxMWEpoYm1Ob01SSXdFQVlEVlFRREV3bGpZWFIwYkdVdFkyRXdIaGNOTVRrdwpOVEE0TURReE5ETTFXaGNOTWpBd05UQTNNRFF4TkRNMVdqQXdNUlV3RXdZRFZRUUtFd3hqWlhKMExXMWhibUZuClpYSXhGekFWQmdOVkJBTVREbkpoYm1Ob1pYSXViWGt1YjNKbk1JSUJJakFOQmdrcWhraUc5dzBCQVFFRkFBT0MKQVE4QU1JSUJDZ0tDQVFFQXJWVjJaUzFSVWFaMG9XWDNORnVRUVRvZFUzTExqL3dqN3JDdXZTMEtWeGlIMWp3TQoxd1VwYmlYU0VZSzRhNXVYRUlPcWZ2UVhYcVhMSTFGWU8wQUswYzkxdmZCUDdUWE13MXRrc01nRXRmUmNaMkt1CllleE5HajJEdDYrbUEvVk9FcVZsc2x1SDd5c0FPYllyYXQ0RmdVNmtWOUpQeGtzVzNmRys5ekVzZmIvSGh6TXAKVHFJVkh1OE1PbVhubjVHeG9qZDdHTGRTTlRyUmllVUh0Y2d4M0tmVW1TVEFHaFAvZjhrL2RMT0ZHRjlVd2J2ZApqa2xsUzBnV3VFSk12cklYamU1ek53Ylc2Q2Z4QjJuQ3VEMHlSbDRXcmlYZmN1T1hPV2ZtbjQrVmpsM2ptdmRvCkhRVjh2UXVNcXhKdm5xTkFuckN5Q3RTWTQyU3lZZnFSRXk4QkxRSURBUUFCb3pzd09UQU9CZ05WSFE4QkFmOEUKQkFNQ0JhQXdEQVlEVlIwVEFRSC9CQUl3QURBWkJnTlZIUkVFRWpBUWdnNXlZVzVqYUdWeUxtMTVMbTl5WnpBTgpCZ2txaGtpRzl3MEJBUXNGQUFPQ0FRRUFveVpTVWM5bGRTcktybEFCQy9DNzA4bXczcVJaWEp3RmZ3K1NtTDkwCmRzTHR0ZU5OWnhNUDZvWDFISHcyYml4Yy9wbE4xRVJhYisxbWJ1NUJ0RW1zY1d0ZFJZbDc3dDJUcHh1VkxhY0wKRHlMcFhOUXRPc1NHN25mM2tMeU5kdjlhNnc0SXpTcUhCUnZCWGdHRktVSDdVWG1aM0hLWEFxaHE3N0RoTC9VRgpxYkVWQm02SUVyT3R2cjB0T3N5WEpOdHNGelhEVFFTVGNCcEszSkQ5akg5Q25nbHdjRHk1a0NXd2JwQVh4bGhhCmJiMTJSK0w1WCtNdVNMbEpFVUcybitJakZHS08xSG9OTGlPZ1pDdFArbm1BTXpRYTR3YTc0MDNRcEhRd3ptTEEKMm1Hbmx0bXFuYW5CcnBXcGREa2puZkxRYk9wZzNqLzFOV3M2TGxzRk02V1VnUT09Ci0tLS0tRU5EIENFUlRJRklDQVRFLS0tLS0KLS0tLS1CRUdJTiBDRVJUSUZJQ0FURS0tLS0tCk1JSUM3akNDQWRhZ0F3SUJBZ0lCQURBTkJna3Foa2lHOXcwQkFRc0ZBREFvTVJJd0VBWURWUVFLRXdsMGFHVXQKY21GdVkyZ3hFakFRQmdOVkJBTVRDV05oZEhSc1pTMWpZVEFlRncweE9UQTFNRGd3TkRFek5UQmFGdzB5T1RBMQpNRFV3TkRFek5UQmFNQ2d4RWpBUUJnTlZCQW9UQ1hSb1pTMXlZVzVqYURFU01CQUdBMVVFQXhNSlkyRjBkR3hsCkxXTmhNSUlCSWpBTkJna3Foa2lHOXcwQkFRRUZBQU9DQVE4QU1JSUJDZ0tDQVFFQTFyYkdUL1grRXZ4TWtxaEUKYTQxYnVQdE5YL3pmTFZiUndZYTFVVmp3QXJPc1ZNTEJWMjg5Q1c0VGR6UHJLdUVBMTA1NUdKZlRKTEJ2aThqYgpEdCtrbTcvYTk1K2Y5Vm5aaDZaT2VFY0tiaHdnei9lRDJuS2VZVUlPN0JhSEV1VktjQnBsd3RjUUFKVWRUZTl6CjFRVkFGU2l4aThMeW5KSDlSWTRtZWhDcmh1N2szMlA1OFFUQWxrNVA3aFE5ZGYrTkx1UWhaVjhPa3VxQTNRb0kKTGtpQVhoRkVrRWhDTVpLcy8rU2JkRlZxeHRIbXRFSjJtc2I2M2oxdjltcld6WkhEbmhQS3BoNkJ5aW0ralBiNQpJUFltSUJnRkZRRGlqVVljN0RYaG1ZQjgwbXZlbmQ3eDFHamFWUzgxSllueWoyQlRQQlg5cFMzN2JsMTgyd2FBClFFczhSd0lEQVFBQm95TXdJVEFPQmdOVkhROEJBZjhFQkFNQ0FxUXdEd1lEVlIwVEFRSC9CQVV3QXdFQi96QU4KQmdrcWhraUc5dzBCQVFzRkFBT0NBUUVBMWIxVFVYUjNKNDNuU0p1azdOY3lDdTVRQXdIR3lkTWFqeTVwMGZRbgpwUE80d0tEaW1VQnQ2RHRVTGpwQUJML25GYWhCY3lpdEdoNDB1SXpIdHZiRFdkczZ5cFd5S2ZPY2Nub25mTzA1CnBxVDlzS0I4L1F2aldDWXlHODQ1akRpUFdmeUdpMHRySXRLMC9UaStDbENXaTlNYjg0UlpYT2VBbjR3SHVaTjQKU3pCWGJ2MUJIelVrNHZRelhVVjE1YXJRRU1CcmUvQkkzSkVIT3kxRFlaYmk2MHFtekN1RzhheVl2eXA2VEFXOApwWXdsSklZZlI3ZHBjdUJqWnpvNmlEb2NxZFlQZkhrVklIcHZ4am9EZGpXYjcwbEkwYW9Ra3RPeExoc2oreE9aCmVnWTdFOFJsWDNkdmlRRW51VU5EdGhvSEw2USs3Y0ZqVWtTSk9OdUhpMzUzR1E9PQotLS0tLUVORCBDRVJUSUZJQ0FURS0tLS0tCg==
  tls.key: LS0tLS1CRUdJTiBSU0EgUFJJVkFURSBLRVktLS0tLQpNSUlFcFFJQkFBS0NBUUVBclZWMlpTMVJVYVowb1dYM05GdVFRVG9kVTNMTGovd2o3ckN1dlMwS1Z4aUgxandNCjF3VXBiaVhTRVlLNGE1dVhFSU9xZnZRWFhxWExJMUZZTzBBSzBjOTF2ZkJQN1RYTXcxdGtzTWdFdGZSY1oyS3UKWWV4TkdqMkR0NittQS9WT0VxVmxzbHVIN3lzQU9iWXJhdDRGZ1U2a1Y5SlB4a3NXM2ZHKzl6RXNmYi9IaHpNcApUcUlWSHU4TU9tWG5uNUd4b2pkN0dMZFNOVHJSaWVVSHRjZ3gzS2ZVbVNUQUdoUC9mOGsvZExPRkdGOVV3YnZkCmprbGxTMGdXdUVKTXZySVhqZTV6TndiVzZDZnhCMm5DdUQweVJsNFdyaVhmY3VPWE9XZm1uNCtWamwzam12ZG8KSFFWOHZRdU1xeEp2bnFOQW5yQ3lDdFNZNDJTeVlmcVJFeThCTFFJREFRQUJBb0lCQUdiaU4yaE93ZG5wSjRrMQpjTW9QSDB6WUpkZ0daM1dMZi9pSnZ1NWk4YVJJMXpmK1pOQ3NLNjlKbTZyeElHNGU2RWZSSHo1RHZDdGQxWG1kCkt2WGdTWXZIekdkMVlMdTRGMVpOeFRKNVJ2S0puTjl3MzNrWVBadTVyNWswRlNKOFZGOFU5WXdzY0Q4eUV1d3UKaVE3aHdmckNweUhKbjdoZmthOGpjdXBlc2IzaFFZZTArbkJYNHNTd3RvMFJLWThLaDUxaFNucVl2ekRLa2l1bApUenpCb0VuK0tlQ0o5RzlkRG5CQjZtcVRhUlpMUGYwL2NuYUtWQ0kvUThYMUx1OVFQVG9iWkNtcEoraEZzalVsCjZlR2hsUGMrK25FcGN4c3hyeUprMENuV0lZWDU3WGRyaG1HbFEzSVNLNHJlayt5VkVYa2lBLyt6SWlnelIzTkYKcWs1bm93RUNnWUVBMGxIdVpGbFlzSkR5TDhxdWRXM2dzUHhQRzJKR0Y1bEdINnRLT0VmSGN1R1pTMHZMV0NLMwpCYlZaaU81MXByUStobUVnUzY1OTFLWU4xbUVNam9ORGNnQXJlZ1ZDVTg0VDRlcXl5OGxpc0l5emV3UDFUdHM1CkcvcEtHSnNycVMvSVVNVC83eFJQVXR3c3hlQmR2cUlpZWZnT0JlUVJYVldxTnplNHZBOHhsOTBDZ1lFQTB2c00KeW1ZRVJhV0dWRnM5REllamJhRXdsTThJcjFoNTM5RUdaMGRham1wa0NZalE2UVlSaFFKZlVtbE10N1hJWWQzego0M0JjWWE0R0VHV1RsUHExQ0lRbU9kYklsUDRqWFB4alhjSFgyM1ZBUDVVMTAreTlWeUV2dmFjT2xkNVB2V3RFCjlXRm1Mek5BNEFlVVBlNFk2aVc3Q253cHdJVWpMSENXV1FlT29aRUNnWUVBcjBuajB6VDRDeWl6QVF2K3ZSWlgKeDRweDhCUEc4VklvdEtwUlVIamFqUy9qVUhNYVJobWd5cHdyeEVsaU93OU1jVGt2N29PK2tlWFg4STJ3Sm5MVwpXb1F6dUIzNyt0YnRrVVBId3JyVWl1cXBoK1J6cGtabDRhQ0g4dHdZMW9oaGZwZkI2eVBWUE8wT1Q5bEFxR3AvCks0S0hSWGVnMVZaWmFHNERoSkZ4TnUwQ2dZRUFwdXpTZGVDNVNKRnA4cHd0ZFJoc0NYVWJCdUczTVdTREtMNHMKbnRJcHA3d3RZZzNKbkZTY3ordk1sMXVRdzFra0xhcDZLRnpoK3JiQlBYL2dnR0xMa08xZGdTMmFqYWQxTzBERgpMWTBNeUNuQXA0M2FoTTR1RUIzRnMvdlU0bVorR0pRaWZ6S01zcE9PcExIYWJCVlJ5UkFKS0lhTVVqSUVYa2pGCjY3SzYrc0VDZ1lFQWkyMEhTWGwzK2p1UG11OXBWNDBmbDdtbllLUEkwUVZBUDNqdThwdWg2cWN2SmJCUFBHMVkKRzNmeHRDcnlXU1BEbGxGbFhtMlN1ZkpVQUI3QkdRZjBsWjJGZTg4V3hVRHQ2Y1NYbXhwL3RGemhpMmFqTGNkbQpEd3dWYlZJaHhSYVIyVThUNTZsclNXaE1kSFpYWjVKVXp6NStnTnNkT21mZWFpQkJLdURPUHQ4PQotLS0tLUVORCBSU0EgUFJJVkFURSBLRVktLS0tLQo=
kind: Secret
metadata:
  annotations:
    certmanager.k8s.io/alt-names: rancher.my.org
    certmanager.k8s.io/common-name: rancher.my.org
    certmanager.k8s.io/issuer-kind: Issuer
    certmanager.k8s.io/issuer-name: rancher
  creationTimestamp: 2019-05-08T04:14:35Z
  labels:
    certmanager.k8s.io/certificate-name: tls-rancher-ingress
  name: tls-rancher-ingress
  namespace: cattle-system
  resourceVersion: "8671"
  selfLink: /api/v1/namespaces/cattle-system/secrets/tls-rancher-ingress
  uid: c9a726b7-7147-11e9-ade3-5254009b5518
type: kubernetes.io/tls

将data中的tls.crt 、tls.key用base解码至正常格式，添加进证书中

在ingress界面点击更新转发配置，勾选https:443将会弹出服务器证书的选择框，此时我们点击现在创建，进入到证书管理界面

解码

将data中的tls.crt 、tls.key用base64解码至正常格式，添加进证书中

填写证书信息

修改ingress配置

回到ingress界面，选择上一步创建的证书，配置转发规则，由于是测试的搭建，所以域名这里也不填写了。

访问rancher

在修改完ingress后，vip已创建出来

通过访问vip 111.230.121.109，可以看到浏览器会提示不是安全的链接，因为这里没有用域名访问且没有对域名等进行配置，在一开始install rancher时 hostname=rancher.my.org，所以后续如果有域名需求的，在一开始install时就要把域名等信息配置好。

继续访问就可以看到进入rancher的界面了，首次访问需要给admin设置密码

设置你的URL，这里是测试所以不配置域名了。

进去后就可以看到rancher的界面了，底下还可以选择中文界面。

通过rancher安装prometheus grafana

点击集群中，点击启用监控并查看实时监控指标

保存配置后，将会创建相关的pod，由于选择了持久化存储，也会创建对应的pvc。

$ kubectl get pod --all-namespaces
NAMESPACE           NAME                                                       READY     STATUS    RESTARTS   AGE
cattle-prometheus   exporter-kube-state-cluster-monitoring-565c8854d7-56x8m    1/1       Running   0          2m
cattle-prometheus   exporter-node-cluster-monitoring-2x9hn                     1/1       Running   0          2m
cattle-prometheus   exporter-node-cluster-monitoring-d5p9d                     1/1       Running   0          2m
cattle-prometheus   exporter-node-cluster-monitoring-kf8q2                     1/1       Running   0          2m
cattle-prometheus   exporter-node-cluster-monitoring-kqspd                     1/1       Running   0          2m
cattle-prometheus   exporter-node-cluster-monitoring-lrqnw                     1/1       Running   0          2m
cattle-prometheus   exporter-node-cluster-monitoring-x2ncf                     1/1       Running   0          2m
cattle-prometheus   grafana-cluster-monitoring-65c9c7c69f-g2mhb                2/2       Running   0          2m
cattle-prometheus   prometheus-cluster-monitoring-0                            5/5       Running   1          1m
cattle-prometheus   prometheus-operator-monitoring-operator-6d8c95d9c6-vgnp8   1/1       Running   0          2m
cattle-system       cattle-cluster-agent-7b5b9765b4-x6nk2                      1/1       Running   0          2m
cattle-system       cattle-node-agent-hjwrg                                    1/1       Running   0          2m
cattle-system       cattle-node-agent-hs7xw                                    1/1       Running   0          2m
cattle-system       cattle-node-agent-lbh6k                                    1/1       Running   0          2m
cattle-system       rancher-7bc85fd5ff-9gkpp                                   1/1       Running   0          27m
cattle-system       rancher-7bc85fd5ff-d2mms                                   1/1       Running   1          27m
cattle-system       rancher-7bc85fd5ff-mp696                                   1/1       Running   1          27m
$ kubectl get sc
NAME            PROVISIONER                    AGE
cbs (default)   cloud.tencent.com/qcloud-cbs   34m
$ kubectl get pvc -n cattle-prometheus
NAME                                                               STATUS    VOLUME                                     CAPACITY   ACCESS MODES   STORAGECLASS   AGE
grafana-cluster-monitoring                                         Bound     pvc-9f243bf7-716a-11e9-9473-525400d5a795   10Gi       RWO            cbs            3m
prometheus-cluster-monitoring-db-prometheus-cluster-monitoring-0   Bound     pvc-abe75969-716a-11e9-82b2-525400489be9   50Gi       RWO            cbs            3m

此时再回到集群界面可以看到界面上多了grafana的图标，点进去可以跳转到grafana

界面

常见问题：

1.创建prometheus 时，选择了持久化数据，但是广州二区cbs盘售罄了导致pvc创建失败

解决方式：

换个资源的区域
换个持久化方式，默认选的是cbs storageclasses，可以通过创建nfs storageclasses来将数据持久化：TKE创建StorageClass配置Provisioner为nfs

2.ingress不能选择后端服务，因为rancher的service type类型没有修改为nodeport，ingress若要转发至service不能为clusterip type。

3.install rancher-stable/rancher报错

需要先安装cert-manager，请看第四步骤

helm install rancher-stable/rancher  \
> --name rancher \
> --namespace cattle-system \
> --set hostname=rancher.my.org
Error: validation failed: unable to recognize "": no matches for kind "Issuer" in version "certmanager.k8s.io/v1alpha1"

参考链接

https://www.cnrancher.com/docs/rancher/v2.x/cn/overview/

https://rancher.com/docs/rancher/v2.x/en/installation/ha/helm-rancher/

原创声明：本文系作者授权腾讯云开发者社区发表，未经许可，不得转载。

如有侵权，请联系 cloudcommunity@tencent.com 删除。

kubernetes

容器服务

原创声明：本文系作者授权腾讯云开发者社区发表，未经许可，不得转载。

如有侵权，请联系 cloudcommunity@tencent.com 删除。

kubernetes

容器服务

登录后参与评论

0 条评论

热度