Ingress-Nginx 服务暴露基础学习与实践 (2)

原创

全栈工程师修炼指南

修改于 2021-07-26 11:17:17

3.2K0

修改于 2021-07-26 11:17:17

文章被收录于专栏：全栈工程师修炼之路

如下基础示例请看上一篇文章

示例1.Ingress 常规使用方案

示例2.Ingress HTTPS 代理访问

示例3.Ingress Rewrite 重写重定向访问

示例4.Ingress VirtualHost 虚拟主机访问

示例5.Ingress BasicAuth（基础认证）功能

示例6.Ingress Redirect 域名重定向功能

示例7.Ingress 黑白名单访问限制

更多学习笔记文章请关注 WeiyiGeek 公众账号，学习交流【邮箱联系: Master#weiyigeek.top】

原文地址: https://mp.weixin.qq.com/s/ZxyNCP32BQjQo5ygztJttQ

示例8.Ingress Header 请求头匹配

描述: Ingress-nginx 的匹配请求头，可以采用nginx.ingress.kubernetes.io/server-snippet注解。

# - 资源清单

tee ingress-header-request.yaml <<'EOF'

apiVersion: networking.k8s.io/v1

kind: Ingress

metadata:

  name: ingress-heade

  namespace: demo

  annotations:

    nginx.ingress.kubernetes.io/server-snippet: |-

      set $agentflag 0;

      if ($http\_user\_agent ~\* "(Firefox)" ){

        set $agentflag 1;

      }

      if ($agentflag = 1) {

        return 301 https://www.weiyigeek.top;

      }

spec:

  rules: 

    - host: header.weiyigeek.top

      http:

        paths:

        - path: /

          pathType: ImplementationSpecific

          backend:

            service:

              name: backend-svc

              port:

                number: 8080

status:

  loadBalancer: {}

EOF



# - 创建查看 Ingress 资源对象

kubectl create -f ingress-header-request.yaml

kubectl get ing -n demo ingress-header && kubectl describe ing -n demo ingress-heade

  # Rules:

  #   Host                  Path  Backends

  #   ----                  ----  --------

  #   header.weiyigeek.top  /     backend-svc:8080 (192.168.109.90:8080,192.168.109.93:8080)



# - 验证 Ingress 资源对象

# (1) 指定 User-Agent 请求头 包含 Firefox

$ curl http://10.10.107.221/ -H 'host: header.weiyigeek.top' -H 'User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:89.0) Gecko/20100101 Firefox/89.0' -k -i

  # HTTP/1.1 301 Moved Permanently

  # Date: Fri, 16 Jul 2021 14:00:04 GMT

  # Content-Type: text/html

  # Content-Length: 162

  # Connection: keep-alive

  # Location: https://www.weiyigeek.top



  # <html>

  # <head><title>301 Moved Permanently</title></head>

  # <body>

  # <center><h1>301 Moved Permanently</h1></center>

  # <hr><center>nginx</center>

  # </body>

  # </html>



# (2) 指定 User-Agent 请求头 不包含 Firefox 

$ curl http://10.10.107.221/ -H 'host: header.weiyigeek.top' -H 'User-Agent: Mozilla/5.0' -k -i

  # HTTP/1.1 200

  # Date: Fri, 16 Jul 2021 14:03:29 GMT

  # Content-Type: text/html

  # Content-Length: 40

  # Connection: keep-alive

  # Accept-Ranges: bytes

  # ETag: W/"40-1626351892000"

  # Last-Modified: Thu, 15 Jul 2021 12:24:52 GMT



  # hostname-web-backend-0-WeiyiGeek-Tomcat

示例9.Ingress 灰度金丝雀发布

描述: 在某些情况下，您可能希望通过向与生产服务不同的服务发送少量请求来“金丝雀”一组新的更改。金丝雀注解使 Ingress 规范能够根据应用的规则充当路由请求的替代服务,在nginx.ingress.kubernetes.io/canary: "true"设置后可以启用以下用于配置金丝雀的注释：。

Nginx Annotations 支持以下 4 种 Canary 规则:

\* nginx.ingress.kubernetes.io/canary-weight：基于服务权重的流量切分，适用于蓝绿部署，权重范围 0 - 100 按百分比将请求路由到 Canary Ingress 中指定的服务。权重为 0 意味着该金丝雀规则不会向 Canary 入口的服务发送任何请求。权重为 100 意味着所有请求都将被发送到 Canary 入口。

\* nginx.ingress.kubernetes.io/canary-by-header：基于 Request Header 的流量切分，适用于灰度发布以及 A/B 测试。当 Request Header 设置为 always时，请求将会被一直发送到 Canary 版本；当 Request Header 设置为 never时，请求不会被发送到 Canary 入口；对于任何其他 Header 值，将忽略 Header，并通过优先级将请求与其他金丝雀规则进行优先级的比较。

\* nginx.ingress.kubernetes.io/canary-by-header-value：要匹配的 Request Header 的值，用于通知 Ingress 将请求路由到 Canary Ingress 中指定的服务。当 Request Header 设置为此值时，它将被路由到 Canary 入口。该规则允许用户自定义 Request Header 的值，必须与上一个 annotation (即：canary-by-header）一起使用。

\* nginx.ingress.kubernetes.io/canary-by-header-pattern：这与canary-by-header-valuePCRE 正则表达式匹配的工作方式相同。请注意，canary-by-header-value设置此注释时将被忽略。当给定的 Regex 在请求处理过程中导致错误时，该请求将被视为不匹配。

\* nginx.ingress.kubernetes.io/canary-by-cookie：基于 Cookie 的流量切分，适用于灰度发布与 A/B 测试。用于通知 Ingress 将请求路由到 Canary Ingress 中指定的服务的cookie。当 cookie 值设置为 always时，它将被路由到 Canary 入口；当 cookie 值设置为 never时，请求不会被发送到 Canary 入口；对于任何其他值，将忽略 cookie 并将请求与其他金丝雀规则进行优先级的比较。 定义两个版本的代码。

Tips : 金丝雀规则是按优先顺序计算的。优先级如下：canary by header->canary by cookie->canary weight

Tips : 注意当您将入口标记为 Canary 时，除nginx.ingress.kubernetes.io/load-balanceand之外的所有其他非 Canary 注释都将被忽略（从相应的主入口继承）nginx.ingress.kubernetes.io/upstream-hash-by。`

**资源配置清单:**

# - 默认访问前端

tee ingress-canary-v1.yaml <<'EOF'

apiVersion: networking.k8s.io/v1

kind: Ingress

metadata:

  name: ingress-canary-v1

  namespace: demo

  annotations:

    nginx.ingress.kubernetes.io/canary: "true"

    nginx.ingress.kubernetes.io/canary-by-header: canary

    nginx.ingress.kubernetes.io/canary-by-header-value: v1

    nginx.ingress.kubernetes.io/canary-weight: "50"

spec:

  rules: 

    - host: canary.weiyigeek.top

      http:

        paths:

        - path: /

          pathType: ImplementationSpecific

          backend:

            service:

              name: front-svc

              port:

                number: 80

EOF



# - 利用金丝雀访问后端

tee ingress-canary-v2.yaml <<'EOF'

apiVersion: networking.k8s.io/v1

kind: Ingress

metadata:

  name: ingress-canary-v2

  namespace: demo

  annotations:

    nginx.ingress.kubernetes.io/canary: "true"

    nginx.ingress.kubernetes.io/canary-by-header: canary

    nginx.ingress.kubernetes.io/canary-by-header-value: v2

    nginx.ingress.kubernetes.io/canary-weight: "50"

spec:

  rules: 

    - host: canary.weiyigeek.top

      http:

        paths:

        - path: /

          pathType: ImplementationSpecific

          backend:

            service:

              name: backend-svc

              port:

                number: 8080

EOF



~/k8s/ingress/demo1# kubectl create -f ingress-canary-v1.yaml

  # ingress.networking.k8s.io/ingress-canary-v1 created

~/k8s/ingress/demo1# kubectl create -f ingress-canary-v2.yaml

  # ingress.networking.k8s.io/ingress-canary-v2 created



~/k8s/ingress/demo1# kubectl get ing -n demo ingress-canary-v1 ingress-canary-v2

  # NAME                CLASS    HOSTS                  ADDRESS   PORTS   AGE

  # ingress-canary-v1   <none>   canary.weiyigeek.top             80      3m12s

  # ingress-canary-v2   <none>   canary.weiyigeek.top             80      3m4s



# 验证金丝雀 Ingress 对象应用

for i in {1..3};do curl http://10.10.107.221/ -H 'host: canary.weiyigeek.top' -i;done

for i in {1..3};do curl http://10.10.107.221/ -H 'host: canary.weiyigeek.top' -H 'weiyigeek: v2' -i;done

示例10.Ingress 相关配置补充

**(1) 自定义错误页面**

参考地址: https://github.com/kubernetes/ingress-nginx/blob/master/docs/examples/customization/custom-errors/custom-default-backend.yaml

描述: 可以修改 deploy.yaml 中指定 nginx-ingress-controller 运行的参数，在此处添加- --default-backend-service=名称空间/SVC名称参数

$ grep -A 3 -n "/nginx-ingress" deploy.yaml

332:            - /nginx-ingress-controlle

333-            - --publish-service=$(POD\_NAMESPACE)/ingress-nginx-controlle

334-            - --election-id=ingress-controller-leade

335-            - --ingress-class=nginx

                - --default-backend-service=demo/error-svc

**(2) 连接和传输速率限制**

描述: 下述注释定义了对连接和传输速率的限制，这些可以用来减轻DDoS攻击。

\* nginx.ingress.kubernetes.io/limit-connections：单个IP地址允许的并发连接数。超出此限制时，将返回503错误。

\* nginx.ingress.kubernetes.io/limit-rps：每秒从给定IP接受的请求数。突发限制设置为此限制乘以突发乘数，默认乘数为5。当客户端超过此限制时，将 返回limit-req-status-code默认值： 503。

\* nginx.ingress.kubernetes.io/limit-rpm：每分钟从给定IP接受的请求数。突发限制设置为此限制乘以突发乘数，默认乘数为5。当客户端超过此限制时，将 返回limit-req-status-code默认值： 503。

\* nginx.ingress.kubernetes.io/limit-burst-multiplier：突发大小限制速率的倍数。默认的脉冲串乘数为5，此注释将覆盖默认的乘数。当客户端超过此限制时，将 返回limit-req-status-code默认值： 503。

\* nginx.ingress.kubernetes.io/limit-rate-after：最初的千字节数，在此之后，对给定连接的响应的进一步传输将受到速率的限制。必须在启用代理缓冲的情况下使用此功能。

\* nginx.ingress.kubernetes.io/limit-rate：每秒允许发送到给定连接的千字节数。零值禁用速率限制。必须在启用代理缓冲的情况下使用此功能。

\* nginx.ingress.kubernetes.io/limit-whitelist：客户端IP源范围要从速率限制中排除。该值是逗号分隔的CIDR列表。

**示例说明:**

# - 资源清单配置:

tee ingress-limit.yaml <<'EOF'

apiVersion: networking.k8s.io/v1

kind: Ingress

metadata:

  name: ingress-limit

  namespace: demo

  annotations:

    nginx.ingress.kubernetes.io/limit-rate: "100K"               # 限制客户端每秒传输的字节数

    nginx.ingress.kubernetes.io/limit-whitelist: "10.10.107.220" # 白名单中的IP不限速

    nginx.ingress.kubernetes.io/limit-rps: 1                     # 单个IP每秒的连接数

    nginx.ingress.kubernetes.io/limit-rpm: 30                    # 单个IP每分钟的连接数

spec:

  rules: 

    - host: limit.weiyigeek.top

      http:

        paths:

        - path: /

          pathType: ImplementationSpecific

          backend:

            service:

              name: backend-svc

              port:

                number: 8080

status:

  loadBalancer: {}

EOF

**(3) ConfigMap在Ingress Controller中实战**

描述: 前面我们说到我们除了可以采用annotations方式配置ingress规则外也可以通过ConfigMap来存储相应的Ingress规则;

Tips : 通过 Ingress ConfigMaps 查看详细的配置参数。

Ingress 规则存储配置清单：

# --configmap=

tee nginx-config.yaml <<'EOF'

kind: ConfigMap

apiVersion: v1

metadata:

  name: ingress-nginx-controller  # 注意此ConfigMap名称与/nginx-ingress-controller 中 `--configmap=$(POD\_NAMESPACE)/ingress-nginx-controller` 对应

  namespace: ingress-nginx

  labels:

    app: ingress-nginx

data:

  proxy-connect-timeout: "15"   #这里有个坑，到家一定要注意是'-'不是'\_'

EOF



# --tcp-services-configmap=

---

kind: ConfigMap

apiVersion: v1

metadata:

  name: tcp-services

  namespace: ingress-nginx

  labels:

    app.kubernetes.io/name: ingress-nginx

    app.kubernetes.io/part-of: ingress-nginx



# --udp-services-configmap

---

kind: ConfigMap

apiVersion: v1

metadata:

  name: udp-services

  namespace: ingress-nginx

  labels:

    app.kubernetes.io/name: ingress-nginx

    app.kubernetes.io/part-of: ingress-nginx

nginx-ingress-controller 使用 Configmap 作为配置存在控制器:

# 可以在 nginx-ingress-controller 部署资源清单中看见`--configmap=`参数

vim +328 deploy.yaml

 containers:

        - name: controlle

          image: pollyduan/ingress-nginx-controller:v0.47.0

          imagePullPolicy: IfNotPresent

          lifecycle:

            preStop:

              exec:

                command:

                  - /wait-shutdown

          args:

            - /nginx-ingress-controlle

            - --publish-service=$(POD\_NAMESPACE)/ingress-nginx-controlle

            - --election-id=ingress-controller-leade

            - --ingress-class=nginx

            - --configmap=$(POD\_NAMESPACE)/ingress-nginx-controller   # 使用 confimap 配置文件

            - --validating-webhook=:8443

            - --validating-webhook-certificate=/usr/local/certificates/cert

            - --validating-webhook-key=/usr/local/certificates/key

查看ingress-nginx名称空间中的ConfigMap配置

kubectl apply -f nginx-config.yaml

  # configmap/ingress-nginx-controller configured



kubectl get cm -n ingress-nginx

  # NAME                              DATA   AGE

  # ingress-controller-leader-nginx   0      2d8h

  # ingress-nginx-controller          0      2d8h

  # kube-root-ca.crt                  1      2d8h



kubectl describe cm -n ingress-nginx ingress-nginx-controlle

  # Name:         ingress-nginx-controlle

  # Namespace:    ingress-nginx

  # Labels:       app.kubernetes.io/component=controlle

  #               app.kubernetes.io/instance=ingress-nginx

  #               app.kubernetes.io/managed-by=Helm

  #               app.kubernetes.io/name=ingress-nginx

  #               app.kubernetes.io/version=0.47.0

  #               helm.sh/chart=ingress-nginx-3.33.0

  # Annotations:  <none>



  # Data

  # ====

  # proxy-connect-timeout:

  # ----

  # 15

  # Events:

  #   Type    Reason  Age   From                      Message

  #   ----    ------  ----  ----                      -------

  #   Normal  UPDATE  25s   nginx-ingress-controller  ConfigMap ingress-nginx/ingress-nginx-controlle

验证是否在ingress-nginx-controller中配置生效:

kubectl get pod -n ingress-nginx -l app.kubernetes.io/component=controlle

  # NAME                                        READY   STATUS    RESTARTS   AGE

  # ingress-nginx-controller-5df4f79b95-6p8rr   1/1     Running   0          2d8h



# 在 ingress-nginx-controller-5df4f79b95-6p8r 的 Pod 中 查看 nginx.conf 文件中 proxy\_connect\_timeout 参数是否为 15s

root@k8s-master-1:~/k8s/ingress/deployment# kubectl -n ingress-nginx exec -it ingress-nginx-controller-5df4f79b95-6p8rr bash

bash-5.1$ grep "proxy\_connect\_timeout" nginx.conf

          proxy\_connect\_timeout                   15s;   #

企业实战

1.利用Ingress访问后端的Nginx Web和Tomcat后端综合实践。

**实验目标:**

1) 在腾讯云创建test.weiyigeek.top以及demo.weiyigeek.top证书以供Ingress使用。
2) 分别在kubernetes集群中创建Nginx Web与Tomcat 后端应用。
3) 分别使用Ingress控制器创建以SSL访问/为NginxWeb而/tomcatTomcat 后端需要BasicAuth（基础认证）访问功能

**详细流程:**

Step 1.在腾讯云中申请免费的SSL证书(TrustAsia TLS)有效期一年,验证时需要设置域名的TXT解析, 申请地址:https://console.cloud.tencent.com/ssl。

Step 2.分别在kubernetes集群中创建Nginx Web与Tomcat 后端应用以及创建名称空间。

tee sts-svc-nginx-tomcat.yaml<<'EOF'

apiVersion: v1

kind: Namespace

metadata:

  name: demo

---

# Service 服务创建(服务是软件服务的命名抽象)。

# kubectl api-resources | grep "service"

apiVersion: v1

kind: Service

metadata:

  name: front-svc

  namespace: demo

  labels:

    app.kubernetes.io/name: ingress-nginx

    app.kubernetes.io/part-of: ingress-nginx

spec:

  selector:

    app: nginx-web

  ports:

    - port: 80 

      targetPort: 80 

      protocol: TCP 

---

apiVersion: v1

kind: Service

metadata:

  name: backend-svc

  namespace: demo

  labels:

    app.kubernetes.io/name: ingress-nginx

    app.kubernetes.io/part-of: ingress-nginx

spec:

  selector:

    app: tomcat-web

  ports:

    - port: 8080

      targetPort: 8080 

      protocol: TCP

  sessionAffinity: None

---

apiVersion: apps/v1

kind: StatefulSet

metadata:

  name: web-front

  namespace: demo

spec:

  replicas: 2

  serviceName: front-svc

  selector:

    matchLabels:

      app: nginx-web

  template:

    metadata:

      labels:

        app: nginx-web

    spec:

      initContainers:

      - name: init-html

        image: busybox:latest

        imagePullPolicy: IfNotPresent

        command: ['sh', '-c', "echo hostname-${HOSTNAME}-${MSG} > /usr/share/nginx/html/index.html"]

        env:

        - name: MSG

          value: "WeiyiGeek-Nginx"

        volumeMounts:

        - name: web

          mountPath: "/usr/share/nginx/html"

        securityContext:

          privileged: true

      containers:

      - name: nginx   

        image: nginx:latest

        imagePullPolicy: IfNotPresent

        env:

        - name: TZ

          value: "Asia/Shanghai"

        ports:

        - name: nginx

          containerPort: 80

        volumeMounts:

        - name: web

          mountPath: "/usr/share/nginx/html"

      volumes:

      - name: web 

        emptyDir: {}

---

apiVersion: apps/v1

kind: StatefulSet

metadata:

  name: web-backend

  namespace: demo

spec:

  replicas: 2

  serviceName: backend-svc

  selector:

    matchLabels:

      app: tomcat-web

  template:

    metadata:

      labels:

        app: tomcat-web

    spec:

      initContainers:

      - name: init-html

        image: busybox:latest

        imagePullPolicy: IfNotPresent

        command: ['sh', '-c', "echo hostname-${HOSTNAME}-${MSG} > /usr/local/tomcat/webapps/ROOT/index.html"]

        env:

        - name: MSG

          value: WeiyiGeek-Tomcat

        volumeMounts:

        - name: web

          mountPath: "/usr/local/tomcat/webapps/ROOT/"

        securityContext:

          privileged: true

      containers:

      - name: nginx   

        image: "tomcat:8.5.69-jdk8-openjdk-slim-buster"

        imagePullPolicy: IfNotPresent

        env:

        - name: TZ

          value: "Asia/Shanghai"

        ports:

        - name: nginx

          containerPort: 8080

        volumeMounts:

        - name: web

          mountPath: "/usr/local/tomcat/webapps/ROOT/"

      volumes:

      - name: web 

        emptyDir: {}

EOF



# 利用资源清单进行创建nginx、Tomcat应用

kubectl create -f sts-svc-nginx-tomcat.yaml

  # namespace/demo created

  # service/front-svc created

  # service/backend-svc created

  # statefulset.apps/web-front created

  # statefulset.apps/web-backend created

Step 3.验证SVC服务负载均衡和请求查看创建的应用

# 查看 Service 控制器对象

$ kubectl get svc -n demo -o wide

  # NAME          TYPE        CLUSTER-IP   EXTERNAL-IP   PORT(S)    AGE   SELECTOR

  # backend-svc   ClusterIP   None         <none>        8080/TCP   12m   name=tomcat-web

  # front-svc     ClusterIP   None         <none>        80/TCP     12m   name=nginx-web



# 查看 StatefulSet 控制器对象

$ kubectl get sts -n demo -o wide

  # NAME          READY   AGE   CONTAINERS   IMAGES

  # web-backend   2/2     13m   nginx        tomcat:8.5.69-jdk8-openjdk-slim-buste

  # web-front     2/2     13m   nginx        nginx:latest



# 查看 nginx 与 tomcat 相关 Pod 信息

$ kubectl get pod -n demo -o wide --show-labels

  # NAME            READY   STATUS    RESTARTS   AGE   IP               NODE        LABELS

  # web-backend-0   1/1     Running   0          13m   192.168.109.90   k8s-node-1  app=tomcat-web,controller-revision-hash=web-backend-776459fc9d,statefulset.kubernetes.io/pod-name=web-backend-0

  # web-backend-1   1/1     Running   0          13m   192.168.109.93   k8s-node-1  app=tomcat-web,controller-revision-hash=web-backend-776459fc9d,statefulset.kubernetes.io/pod-name=web-backend-1

  # web-front-0     1/1     Running   0          13m   192.168.109.91   k8s-node-1  app=nginx-web,controller-revision-hash=web-front-89f7fcc75,statefulset.kubernetes.io/pod-name=web-front-0

  # web-front-1     1/1     Running   0          13m   192.168.109.92   k8s-node-1  app=nginx-web,controller-revision-hash=web-front-89f7fcc75,statefulset.kubernetes.io/pod-name=web-front-1



# 查看 SVC 服务发现利用CoreDNS进行域名解析

$ nslookup front-svc.demo.svc.cluster.local. 10.96.0.10

  # Server:         10.96.0.10

  # Address:        10.96.0.10#53

  # Name:   front-svc.demo.svc.cluster.local

  # Address: 192.168.109.91

  # Name:   front-svc.demo.svc.cluster.local

  # Address: 192.168.109.92

$ nslookup backend-svc.demo.svc.cluster.local. 10.96.0.10

  # ...

  # Name:   backend-svc.demo.svc.cluster.local

  # Address: 192.168.109.90

  # Name:   backend-svc.demo.svc.cluster.local



# 请求查看 Nginx 与 Tomcat 应用回显。

curl front-svc.demo.svc.cluster.local

  # hostname-web-front-0-WeiyiGeek-Nginx

curl front-svc.demo.svc.cluster.local

  # hostname-web-front-1-WeiyiGeek-Nginx

curl backend-svc.demo.svc.cluster.local:8080

  # hostname-web-backend-1-WeiyiGeek-Tomcat

curl backend-svc.demo.svc.cluster.local:8080

  # hostname-web-backend-0-WeiyiGeek-Tomcat

Step 4.采用 secrets 存储我们创建的证书, 参考地址：https://github.com/kubernetes/ingress-nginx/blob/master/docs/examples/PREREQUISITES.md#tls-certificates

# Tips:  kubectl api-resources  | grep "secret" 可查看apiversions以及namespace是否支持

# 假设你已经将证书上传到k8s集群目录中并解压

$ ls

demo.weiyigeek.top.zip  test.weiyigeek.top.zip



$ unzip demo.weiyigeek.top.zip -d demo.weiyigeek.top

$ unzip test.weiyigeek.top.zip -d test.weiyigeek.top



# 创建 tls 类型的 secret

$ cd /tmp/demo.weiyigeek.top/Nginx

kubectl -n demo create secret tls demoweiyigeektop --key 2\_demo.weiyigeek.top.key --cert 1\_demo.weiyigeek.top\_bundle.crt

  # secret/demoweiyigeektop created



$ cd /tmp/test.weiyigeek.top/Nginx

kubectl -n demo create secret tls testweiyigeektop --key 2\_test.weiyigeek.top.key --cert 1\_test.weiyigeek.top\_bundle.crt

  # secret/testweiyigeektop created

Step 4.创建Ingress控制器的配置清单文件及Ingress对应规则。

tee ingress-demo.yaml <<'EOF'

apiVersion: networking.k8s.io/v1

kind: Ingress

metadata:

  name: ingress-demo

  namespace: demo

  annotations:

    # 指定 Ingress Controller 的类型

    kubernetes.io/ingress.class: "nginx"

    # 指定我们的 rules 的 path 可以使用正则表达式

    nginx.ingress.kubernetes.io/use-regex: "true"

    # 连接超时时间，默认为 5s

    nginx.ingress.kubernetes.io/proxy-connect-timeout: "600"

    # 后端服务器回转数据超时时间，默认为 60s

    nginx.ingress.kubernetes.io/proxy-send-timeout: "600"

    # 后端服务器响应超时时间，默认为 60s

    nginx.ingress.kubernetes.io/proxy-read-timeout: "600"

    # 客户端上传文件，最大大小，默认为 20m

    nginx.ingress.kubernetes.io/proxy-body-size: "10m"

    # URL 重写

    nginx.ingress.kubernetes.io/rewrite-target: /

spec:

  # 默认后端服务指定

  defaultBackend:

    service:

      name: front-svc

      port:

        number: 80

  tls:

  - hosts:

    - test.weiyigeek.top

    secretName: testweiyigeektop

  - hosts:

    - demo.weiyigeek.top

    secretName: demoweiyigeektop

  rules:

  - host: test.weiyigeek.top

    http:

      paths:

      - pathType: Prefix

        path: /tomcat

        backend:

          service:

            name: backend-svc

            port:

              number: 8080

      - pathType: Prefix

        path: /nginx

        backend:

          service:

            name: front-svc

            port:

              number: 80

  - host: demo.weiyigeek.top

    http:

      paths:

      - pathType: Prefix

        path: /

        backend:

          service:

            name: front-svc

            port:

              number: 80

EOF

Step 5.创建Ingress并查看相关信息

$ kubectl create -f ingress-demo.yaml

  # ingress.networking.k8s.io/ingress-demo created  



# 查看 ingress-nginx 的 SVC 

kubectl get svc -n ingress-nginx

  # NAME                                 TYPE           CLUSTER-IP       EXTERNAL-IP   PORT(S)                      AGE

  # ingress-nginx-controller             LoadBalancer   10.104.171.212   <pending>     80:31122/TCP,443:32155/TCP   24h

  # ingress-nginx-controller-admission   ClusterIP      10.99.168.221    <none>        443/TCP                      24h



# 查看 ingress-demo 对象相关信息

$ kubectl get ing -n demo ingress-demo

  # NAME           CLASS    HOSTS                                   ADDRESS   PORTS     AGE

  # ingress-demo   <none>   test.weiyigeek.top,demo.weiyigeek.top             80, 443   24h



$ kubectl describe ing -n demo ingress-demo

  # Name:             ingress-demo

  # Namespace:        demo

  # Address:

  # Default backend:  front-svc:80 (192.168.109.91:80,192.168.109.92:80)

  # TLS:

  #   testweiyigeektop terminates test.weiyigeek.top

  #   demoweiyigeektop terminates demo.weiyigeek.top

  # Rules:

  #   Host                Path  Backends

  #   ----                ----  --------

  #   test.weiyigeek.top /tomcat   backend-svc:8080 (192.168.109.90:8080,192.168.109.93:8080)

  #   demo.weiyigeek.top /         front-svc:80 (192.168.109.91:80,192.168.109.92:80)

  # Annotations:          kubernetes.io/ingress.class: nginx

  #                       nginx.ingress.kubernetes.io/proxy-body-size: 10m

  #                       nginx.ingress.kubernetes.io/proxy-connect-timeout: 600

  #                       nginx.ingress.kubernetes.io/proxy-read-timeout: 600

  #                       nginx.ingress.kubernetes.io/proxy-send-timeout: 600

  #                       nginx.ingress.kubernetes.io/rewrite-target: /

  #                       nginx.ingress.kubernetes.io/use-regex: true

  # Events:               <none>

Step 6.验证Ingress规则配置的应用分别访问 https://demo.weiyigeek.top 与 https://test.weiyigeek.top 站点。

$ curl 'https://10.10.107.221' -H 'host: demo.weiyigeek.top' -i -k

  # HTTP/2 200

  # date: Sat, 17 Jul 2021 04:39:37 GMT

  # content-type: text/html

  # content-length: 37

  # last-modified: Thu, 15 Jul 2021 12:24:55 GMT

  # etag: "60f02917-25"

  # accept-ranges: bytes

  # strict-transport-security: max-age=15724800; includeSubDomains



  # hostname-web-front-1-WeiyiGeek-Nginx



$ curl 'https://10.10.107.221/tomcat' -H 'host: test.weiyigeek.top' -i -k

  # HTTP/2 200

  # date: Sat, 17 Jul 2021 04:39:50 GMT

  # content-type: text/html

  # content-length: 40

  # accept-ranges: bytes

  # etag: W/"40-1626351892000"

  # last-modified: Thu, 15 Jul 2021 12:24:52 GMT

  # strict-transport-security: max-age=15724800; includeSubDomains



  # hostname-web-backend-0-WeiyiGeek-Tomcat

0x02 入坑出坑

问题1.K8s Nginx Ingress Controller 转发 X-Forwarded-Proto 请求头的问题

问题说明: 在 Kubernetes 集群上部署了 Nginx Ingress Controller 最前端用的是阿里云七层负载均衡，部署后发现不能正确转发 X-Forwarded-Proto 请求头，造成 http 重定向到 https 无法正常工作，请问如何解决？

# 问题补充：

# 用下面的命令进入 nginx-ingress 容器

kubectl exec -it daemonset/nginx-ingress -n nginx-ingress cat /etc/nginx/conf.d/production-cnblogs-ingress.conf



# 发现问题是下面的配置引起的

proxy\_set\_header X-Forwarded-Proto $scheme;

问题解决: 终于在 Nginx Ingress Controller 的官方帮助文档 Advanced Configuration with Annotations 中找到一个注解（annotation）解决了这个问题，它就是 nginx.org/redirect-to-https: "true"

Annotation: nginx.org/redirect-to-https

ConfigMap Key redirect-to-https

Description Sets the 301 redirect rule based on the value of the http\_x\_forwarded\_proto header on the server block to force incoming traffic to be over HTTPS. Useful when terminating SSL in a load balancer in front of the Ingress controlle

# 根据服务器块上http\_x\_forwarded\_proto报头的值设置301重定向规则，以强制传入的流量使用HTTPS。在入口控制器前的负载均衡器中终止SSL时非常有用;

操作步骤：

1）在 cnblogs-ingress.yaml 中 annotations 下面添加 nginx.org/redirect-to-https: "true"

apiVersion: extensions/v1beta1 

kind: Ingress

metadata:

  name: cnblogs-ingress

  annotations:

    kubernetes.io/ingress.class: nginx

    nginx.org/redirect-to-https: "true"

spec:

  rules:

  - host: q.cnblogs.com

    http:

      paths:

        - backend:

            serviceName: q-web

            servicePort: 80

2) 更新 ingress 配置

kubectl apply -f cnblogs-ingress.yaml

3) 更新 nginx-ingress

kubectl rollout restart daemonset/nginx-ingress -n nginx-ingress && \

kubectl rollout status daemonset/nginx-ingress -n nginx-ingress

4）查看 inginx 容器中的配置

kubectl exec -it daemonset/nginx-ingress -n nginx-ingress cat /etc/nginx/conf.d/production-cnblogs-ingress.conf



# 发现 proxy\_set\_header X-Forwarded-Proto $scheme; 变成了 proxy\_set\_header X-Forwarded-Proto https; ，并且增加了下面的 http 重定向 https 的配置。



if ($http\_x\_forwarded\_proto = 'http') {

    return 301 https://$host$request\_uri;

}

nginx-ingress 自己完成了基于X-Forwarded-Proto 的 http 重定向到 https 的操作，应用都不需要自己处理了。

Tips ：建议采用 kubernetes/ingress-nginx 而非 nginxinc/kubernetes-ingress : https://github.com/kubernetes/ingress-nginx

问题2:extensions/v1beta1 Ingress is deprecated in v1.14+, unavailable in v1.22+; use networking.k8s.io/v1 Ingress

解决办法：根据当前 Kubernetes 版本中资源组与版本进行选择即可;

~/K8s/Day7/demo1$ kubectl api-resources | grep "ingresses"

ingresses                         ing          extensions                     true         Ingress

ingresses                         ing          networking.k8s.io              true         Ingress



# Warning: extensions/v1beta1 Ingress is deprecated in v1.14+, unavailable in v1.22+; use networking.k8s.io/v1 Ingress

apiVersion: networking.k8s.io/v1 # 注意点否则将报上面的预警

问题3.Error from server (InternalError): error when applying patch:

错误信息:

$ kubectl apply -f demo-ingress.yaml

# 错误1

Resource: "networking.k8s.io/v1, Resource=ingresses", GroupVersionKind: "networking.k8s.io/v1, Kind=Ingress"

Name: "nginx", Namespace: "default"

for: "demo-ingress.yaml": Internal error occurred: failed calling webhook "validate.nginx.ingress.kubernetes.io": Post "https://ingress-nginx-controller-admission.ingress-nginx.svc:443/extensions/v1beta1/ingresses?timeout=30s": x509: certificate is valid for k8s-master002, kubernetes, kubernetes.default, kubernetes.default.svc, kubernetes.default.svc.cluster.local, not 

ingress-nginx-controller-admission.ingress-nginx.svc



# 错误2

kubectl create -f ingress-nginx-http-v1.yaml

  # Error from server (InternalError): error when creating "ingress-nginx-http-v1.yaml": Internal error occurred: failed calling webhook "validate.nginx.ingress.kubernetes.io": an error on the server ("") has prevented the request from succeeding

解决办法: Webhook 删除后重新构建 ingress-nginx-http-v1.yaml 资源清单即可

weiyigeek@ubuntu:~/K8s/Day7/demo2$ kubectl delete -A ValidatingWebhookConfiguration ingress-nginx-admission

  # validatingwebhookconfiguration.admissionregistration.k8s.io "ingress-nginx-admission" deleted



weiyigeek@ubuntu:~/K8s/Day7/demo2$ kubectl create -f ingress-nginx-http-v1.yaml  

  # ingress.networking.k8s.io/nginx-ingress-http created  # 未其他报错正常创建

问题4.查看ingress规则时提示`kubernetes之ingress error: endpoints "default-http-backend" not found`警告

问题信息:

$ kubectl describe ing

Default backend: default-http-backend:80 (<error: endpoints "default-http-backend" not found>)

问题原因: 注意:根据您正在使用的Ingress控制器，您可能需要创建一个default-http-backend服务。没有规则的入口将所有流量发送到一个默认后端。默认后端通常是Ingress控制器的一个配置选项，在您的Ingress资源中没有指定。如果Ingress对象中的主机或路径都不匹配HTTP请求，则流量将被路由到默认后端。
解决办法:

# Ingress 控制器

spec:

  defaultBackend:

    service:

      name: nginx-ingress-v1-svc

      port:

        number: 80

原创声明：本文系作者授权腾讯云开发者社区发表，未经许可，不得转载。

如有侵权，请联系 cloudcommunity@tencent.com 删除。

kubernetes

nginx

负载均衡

原创声明：本文系作者授权腾讯云开发者社区发表，未经许可，不得转载。

如有侵权，请联系 cloudcommunity@tencent.com 删除。

kubernetes

nginx

负载均衡

登录后参与评论

0 条评论

热度