TKE集群中nginx-ingress使用实践
现在k8s上服务暴露方式用的最多就是nginx-ingress，今天我们来讲讲nginx-ingress的具体使用，我们在tke上实践下，如何部署使用nginx-ingress，以及nginx的一些注解功能的使用。
今天我们主要是在腾讯云上的tke上进行实践测试，不过测试的功能也是支持在其他类型的k8s上使用，只要部署了nignx-ingress就行。现在tke上为了方便用户能够快速使用nginx-ingress推了了通过组件的方式的来部署，具体部署方式可以参考文档https://cloud.tencent.com/document/product/457/50503。 由于tke的nginx ingress组件不再维护更新了，不支持增量新建nginx ingress，如果想自己搭建nginx-ingress可以参考https://cloud.tencent.com/document/product/457/47293进行部署或者参考ingress-nginx部署章节安装ingress nginx到集群。 下面我们说下具体的部署以及nginx-ingress一些常用的功能如何进行配置使用。
安装部署ingress-nginx
下载上面的压缩包到客户端（能helm访问k8s集群），然后解压ingress-nginx文件夹到本地。
这里ingress-nginx的版本是4.11.3，支持1.26-1.30版本，chart包镜像默认是registry.k8s.io这个镜像仓库，国内地域是无法拉取的，value.yaml里面的镜像已经换成ccr镜像仓库，可以直接拉取。
解压好之后，可以执行如下命令通过helm安装。
- 处理指定ns的ingress资源
# ingress-nginx安装在ingress-nginx，然后只处理default下的ingress
helm install ingress-nginx ./ingress-nginx --namespace ingress-nginx --create-namespace --set controller.scope.enabled=true --set controller.scope.namespace=default- 处理所有ns的ingress资源
helm install ingress-nginx ./ingress-nginx --namespace ingress-nginx --create-namespace1. https的配置
有的时候我们需要给域名配置ssl证书来进行https的安全访问，首先你需要给域名申请证书，可以到现在的公有云上为域名申请免费的证书，申请后将证书通过secret挂载到k8s集群中
kubectl create secret tls example-ssl --key 2_example.tke.niewx.cn.key --cert 1_example.tke.niewx.cn_bundle.crt然后再在ingress中配置下，这里我们配置下tls这个字段
apiVersion: networking.k8s.io/v1beta1
kind: Ingress
metadata:
annotations:
ingress.cloud.tencent.com/direct-access: "false"
kubernetes.io/ingress.class: nwx-ingress
kubernetes.io/ingress.extensiveParameters: '{"AddressIPVersion":"IPV4"}'
kubernetes.io/ingress.http-rules: "null"
kubernetes.io/ingress.https-rules: '[{"host":"example.tke.niewx.cn","path":"/","backend":{"serviceName":"springboot","servicePort":"8080"}}]'
kubernetes.io/ingress.rule-mix: "true"
nginx.ingress.kubernetes.io/use-regex: "true"
name: example-ingress
namespace: test
spec:
rules:
- host: example.tke.niewx.cn
http:
paths:
- backend:
serviceName: springboot
servicePort: 8080
path: /
tls:
- hosts:
- example.tke.niewx.cn
secretName: example-ssl
status:
loadBalancer:
ingress:
- ip: 81.71.131.235这样浏览器输入example.tke.niewx.cn会通过https访问。
2. 域名登录认证
有时候我们的服务没有提供登录认证，但是有不希望将服务提供给所有的人都能访问，那么可以通过ingress上的认证控制访问。今天我们讲讲常用的2种认证方式。
2.1 基本身份认证
[root@VM-0-13-centos nginx-ingress]# htpasswd -c auth admin
New password:
Re-type new password:
Adding password for user admin
[root@VM-0-13-centos nginx-ingress]# kubectl create secret generic basic-auth --from-file=auth -n test
secret/basic-auth created接下来我们在ingress中配置下这个secret，这里我们配置账号密码是admin/admin
apiVersion: networking.k8s.io/v1beta1
kind: Ingress
metadata:
annotations:
ingress.cloud.tencent.com/direct-access: "false"
kubernetes.io/ingress.class: nwx-ingress
kubernetes.io/ingress.extensiveParameters: '{"AddressIPVersion":"IPV4"}'
kubernetes.io/ingress.http-rules: "null"
kubernetes.io/ingress.https-rules: '[{"host":"example.tke.niewx.cn","path":"/","backend":{"serviceName":"springboot","servicePort":"8080"}}]'
kubernetes.io/ingress.rule-mix: "true"
nginx.ingress.kubernetes.io/auth-realm: Authentication Required - admin
nginx.ingress.kubernetes.io/auth-secret: basic-auth
nginx.ingress.kubernetes.io/auth-type: basic
nginx.ingress.kubernetes.io/use-regex: "true"
name: example-ingress
namespace: test
spec:
rules:
- host: example.tke.niewx.cn
http:
paths:
- backend:
serviceName: springboot
servicePort: 8080
path: /
tls:
- hosts:
- example.tke.niewx.cn
secretName: example-ssl
status:
loadBalancer:
ingress:
- ip: 81.71.131.235
nginx.ingress.kubernetes.io/auth-realm: Authentication Required - admin
nginx.ingress.kubernetes.io/auth-secret: basic-auth
nginx.ingress.kubernetes.io/auth-type: basic我们在ingress中加入这3条注解，再登录就会需要输入账号密码了。
2.2 外部身份验证
有时候我们有自己的鉴权中心，也是可以使用外部身份进行认证的，这里我们采用https://httpbin.org/basic-auth/user/passwd这个作为外部身份，这个默认账号和密码user/passwd
apiVersion: networking.k8s.io/v1beta1
kind: Ingress
metadata:
annotations:
ingress.cloud.tencent.com/direct-access: "false"
kubernetes.io/ingress.class: nwx-ingress
kubernetes.io/ingress.extensiveParameters: '{"AddressIPVersion":"IPV4"}'
kubernetes.io/ingress.http-rules: "null"
kubernetes.io/ingress.https-rules: '[{"host":"example.tke.niewx.cn","path":"/","backend":{"serviceName":"springboot","servicePort":"8080"}}]'
kubernetes.io/ingress.rule-mix: "true"
nginx.ingress.kubernetes.io/auth-url: https://httpbin.org/basic-auth/user/passwd
nginx.ingress.kubernetes.io/use-regex: "true"这里我们只需要配置下面这个注解就会用外部身份认证
nginx.ingress.kubernetes.io/auth-url: https://httpbin.org/basic-auth/user/passwd这里我们通过curl的方式验证下
[root@VM-0-3-centos ~]# curl -k https://example.tke.niewx.cn/ -v -H 'Host: example.tke.niewx.cn'
* About to connect() to example.tke.niewx.cn port 443 (#0)
* Trying 81.71.131.235...
* Connected to example.tke.niewx.cn (81.71.131.235) port 443 (#0)
* Initializing NSS with certpath: sql:/etc/pki/nssdb
* skipping SSL peer certificate verification
* SSL connection using TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256
* Server certificate:
* subject: CN=Kubernetes Ingress Controller Fake Certificate,O=Acme Co
* start date: Dec 28 06:14:11 2020 GMT
* expire date: Dec 28 06:14:11 2021 GMT
* common name: Kubernetes Ingress Controller Fake Certificate
* issuer: CN=Kubernetes Ingress Controller Fake Certificate,O=Acme Co
> GET / HTTP/1.1
> User-Agent: curl/7.29.0
> Accept: */*
> Host: example.tke.niewx.cn
>
< HTTP/1.1 401 Unauthorized
< Date: Wed, 30 Dec 2020 11:36:43 GMT
< Content-Type: text/html
< Content-Length: 172
< Connection: keep-alive
< WWW-Authenticate: Basic realm="Fake Realm"
< Strict-Transport-Security: max-age=15724800; includeSubDomains
<
<html>
<head><title>401 Authorization Required</title></head>
<body>
<center><h1>401 Authorization Required</h1></center>
<hr><center>nginx</center>
</body>
</html>
* Connection #0 to host example.tke.niewx.cn left intact
[root@VM-0-3-centos ~]# curl -k https://example.tke.niewx.cn/ -v -H 'Host: example.tke.niewx.cn' -u 'user:passwd'
* About to connect() to example.tke.niewx.cn port 443 (#0)
* Trying 81.71.131.235...
* Connected to example.tke.niewx.cn (81.71.131.235) port 443 (#0)
* Initializing NSS with certpath: sql:/etc/pki/nssdb
* skipping SSL peer certificate verification
* SSL connection using TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256
* Server certificate:
* subject: CN=Kubernetes Ingress Controller Fake Certificate,O=Acme Co
* start date: Dec 28 06:14:11 2020 GMT
* expire date: Dec 28 06:14:11 2021 GMT
* common name: Kubernetes Ingress Controller Fake Certificate
* issuer: CN=Kubernetes Ingress Controller Fake Certificate,O=Acme Co
* Server auth using Basic with user 'user'
> GET / HTTP/1.1
> Authorization: Basic dXNlcjpwYXNzd2Q=
> User-Agent: curl/7.29.0
> Accept: */*
> Host: example.tke.niewx.cn
>
< HTTP/1.1 200
< Date: Wed, 30 Dec 2020 11:37:14 GMT
< Content-Type: text/plain;charset=UTF-8
< Content-Length: 23
< Connection: keep-alive
< Strict-Transport-Security: max-age=15724800; includeSubDomains
<
* Connection #0 to host example.tke.niewx.cn left intact
you access path is root[root@VM-0-3-centos ~]# curl -k https://example.tke.niewx.cn/ -v -H 'Host: example.tke.niewx.cn' -u 'user:user'
* About to connect() to example.tke.niewx.cn port 443 (#0)
* Trying 81.71.131.235...
* Connected to example.tke.niewx.cn (81.71.131.235) port 443 (#0)
* Initializing NSS with certpath: sql:/etc/pki/nssdb
* skipping SSL peer certificate verification
* SSL connection using TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256
* Server certificate:
* subject: CN=Kubernetes Ingress Controller Fake Certificate,O=Acme Co
* start date: Dec 28 06:10:15 2020 GMT
* expire date: Dec 28 06:10:15 2021 GMT
* common name: Kubernetes Ingress Controller Fake Certificate
* issuer: CN=Kubernetes Ingress Controller Fake Certificate,O=Acme Co
* Server auth using Basic with user 'user'
> GET / HTTP/1.1
> Authorization: Basic dXNlcjp1c2Vy
> User-Agent: curl/7.29.0
> Accept: */*
> Host: example.tke.niewx.cn
>
< HTTP/1.1 401 Unauthorized
< Date: Wed, 30 Dec 2020 11:37:20 GMT
< Content-Type: text/html
< Content-Length: 172
< Connection: keep-alive
* Authentication problem. Ignoring this.
< WWW-Authenticate: Basic realm="Fake Realm"
< Strict-Transport-Security: max-age=15724800; includeSubDomains
<
<html>
<head><title>401 Authorization Required</title></head>
<body>
<center><h1>401 Authorization Required</h1></center>
<hr><center>nginx</center>
</body>
</html>
* Connection #0 to host example.tke.niewx.cn left intact我们分别通过不带用户名密码、正确的用户名密码、和不到用户名密码进行测试，最终只有正确输入用户名和密码才返回200。
3. rewrite
有时候我们需要创建ingress对象来为Controller制定转发规则
apiVersion: networking.k8s.io/v1beta1
kind: Ingress
metadata:
annotations:
ingress.cloud.tencent.com/direct-access: "false"
kubernetes.io/ingress.class: ingress
nginx.ingress.kubernetes.io/use-regex: "true"
name: ingress-v3
namespace: test
spec:
rules:
- host: example.tke.niewx.cn
http:
paths:
- backend:
serviceName: springboot
servicePort: 8080
path: /api
tls:
- hosts:
- example.tke.niewx.cn
secretName: example-ssl表示会将https://example.tke.niewx.cn/api/URL转发到集群中的https://example.tke.niewx.cn/api/URL上，也就是说最后一行的path: /api也会当作请求的一部分，追加到url中。
但是，如果nginx实际的请求地址为https://svc-springboot/api/URL，则会报404，找不到服务。
有的时候我们需要将前端用户访问的路径指向后端真实的路径，这里我们就需要用到ingress的rewite来实现了
apiVersion: networking.k8s.io/v1beta1
kind: Ingress
metadata:
annotations:
ingress.cloud.tencent.com/direct-access: "false"
kubernetes.io/ingress.class: ingress
nginx.ingress.kubernetes.io/rewrite-target: /$2
nginx.ingress.kubernetes.io/use-regex: "true"
name: example-ingress
namespace: test
spec:
rules:
- host: example.tke.niewx.cn
http:
paths:
- backend:
serviceName: springboot
servicePort: 8080
path: /api(/|$)(.*)
tls:
- hosts:
- example.tke.niewx.cn
secretName: example-ssl这里我们会进行如下重定向
- example.tke.niewx.cn/api 重写到 example.tke.niewx.cn/
- example.tke.niewx.cn/api/ 重写到 example.tke.niewx.cn/
- example.tke.niewx.cn/api/v1 重写到 example.tke.niewx.cn/v1
正常我们会打印出是访问的实际路径，由于被重写，导致本应打印的应该是/api/v1，现在打印的是v1，说明真实访问的后端是example.tke.niewx.cn/v1
我们也可以将域名的根目录重写到某一个后端的路径下，这样访问根目录下就会重写到后的真实路径，我们测试下将根路径重写到/api/v2下面
apiVersion: networking.k8s.io/v1beta1
kind: Ingress
metadata:
annotations:
ingress.cloud.tencent.com/direct-access: "false"
kubernetes.io/ingress.class: nwx-ingress
kubernetes.io/ingress.extensiveParameters: '{"AddressIPVersion":"IPV4"}'
kubernetes.io/ingress.http-rules: "null"
kubernetes.io/ingress.https-rules: '[{"host":"example.tke.niewx.cn","path":"/","backend":{"serviceName":"springboot","servicePort":"8080"}}]'
kubernetes.io/ingress.rule-mix: "true"
nginx.ingress.kubernetes.io/app-root: /api/v2
nginx.ingress.kubernetes.io/auth-url: https://httpbin.org/basic-auth/user/passwd
nginx.ingress.kubernetes.io/use-regex: "true"
name: example-ingress
namespace: test
spec:
rules:
- host: example.tke.niewx.cn
http:
paths:
- backend:
serviceName: springboot
servicePort: 8080
path: /
tls:
- hosts:
- example.tke.niewx.cn
secretName: example-ssl
status:
loadBalancer:
ingress:
- ip: 81.71.131.235这里我们测试发现访问根目录会跳转到/api/v2下，浏览器输入也会强制跳转到/api/v2下
[root@VM-0-3-centos ~]# curl -I -k https://example.tke.niewx.cn
HTTP/1.1 302 Moved Temporarily
Date: Wed, 30 Dec 2020 12:24:30 GMT
Content-Type: text/html
Content-Length: 138
Connection: keep-alive
Location: https://example.tke.niewx.cn/api/v2rewrite功能可以让我们进行丰富的访问路径配置，这样方便了我们可以通过不同的路径去访问不通的后端。
4. 访问白名单
有时候我们需要给域名配置下访问白名单，我只希望部分ip可以访问我的服务，这时候需要用到ingress的whitelist-source-range，我们可以通过这个注解来配置我们希望放通访问的ip。下面我们只放通10.0.0.0/24可以访问
apiVersion: networking.k8s.io/v1beta1
kind: Ingress
metadata:
annotations:
ingress.cloud.tencent.com/direct-access: "false"
kubernetes.io/ingress.class: nwx-ingress
kubernetes.io/ingress.extensiveParameters: '{"AddressIPVersion":"IPV4"}'
kubernetes.io/ingress.http-rules: "null"
kubernetes.io/ingress.https-rules: '[{"host":"example.tke.niewx.cn","path":"/","backend":{"serviceName":"springboot","servicePort":"8080"}}]'
kubernetes.io/ingress.rule-mix: "true"
nginx.ingress.kubernetes.io/use-regex: "true"
nginx.ingress.kubernetes.io/whitelist-source-range: 10.0.0.0/24我们测试下看看，我们在10.0.0.3和10.0.5.33分别访问下域名看看
[root@VM-5-33-tlinux ~]# curl -I -k https://example.tke.niewx.cn
HTTP/1.1 403 Forbidden
Date: Wed, 30 Dec 2020 12:39:35 GMT
Content-Type: text/html
Content-Length: 146
Connection: keep-alive
Strict-Transport-Security: max-age=15724800; includeSubDomains
[root@VM-5-33-tlinux ~]# ip addr | grep eth0
2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP group default qlen 1000
inet 10.0.5.33/24 brd 10.0.5.255 scope global eth0[root@VM-0-3-centos ~]# curl -I -k https://example.tke.niewx.cn
HTTP/1.1 200
Date: Wed, 30 Dec 2020 12:36:05 GMT
Content-Type: text/plain;charset=UTF-8
Content-Length: 23
Connection: keep-alive
Strict-Transport-Security: max-age=15724800; includeSubDomains
[root@VM-0-3-centos ~]# ip addr | grep eth0
2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP group default qlen 1000
inet 10.0.0.3/24 brd 10.0.0.255 scope global eth0可以发现10.0.0.3这个服务器上可以访问，但是10.0.5.33确无法访问，这是因为10.0.0.3在我们配置的白名单里。
5. 永久重定向
我们可以在ingress配置域名的重定向，可以配置访问到其他链接，同时也可以配置重定向错误码，这里错误码值的范围是 300~308 ，超出这个范围就重置为默认301的值。
apiVersion: networking.k8s.io/v1beta1
kind: Ingress
metadata:
annotations:
ingress.cloud.tencent.com/direct-access: "false"
kubernetes.io/ingress.class: nwx-ingress
kubernetes.io/ingress.extensiveParameters: '{"AddressIPVersion":"IPV4"}'
kubernetes.io/ingress.http-rules: "null"
kubernetes.io/ingress.https-rules: '[{"host":"example.tke.niewx.cn","path":"/","backend":{"serviceName":"springboot","servicePort":"8080"}}]'
kubernetes.io/ingress.rule-mix: "true"
nginx.ingress.kubernetes.io/permanent-redirect: https://www.baidu.com
nginx.ingress.kubernetes.io/use-regex: "true"这里我们在浏览器中输入example.tke.niewx.cn会强制跳转到百度页面
[root@VM-0-3-centos ~]# curl -I -k https://example.tke.niewx.cn
HTTP/1.1 301 Moved Permanently
Date: Wed, 30 Dec 2020 12:55:21 GMT
Content-Type: text/html
Content-Length: 162
Connection: keep-alive
Location: https://www.baidu.com
Strict-Transport-Security: max-age=15724800; includeSubDomains下面我们重定向到baidu，然后让返回码是308
apiVersion: networking.k8s.io/v1beta1
kind: Ingress
metadata:
annotations:
ingress.cloud.tencent.com/direct-access: "false"
kubernetes.io/ingress.class: nwx-ingress
kubernetes.io/ingress.extensiveParameters: '{"AddressIPVersion":"IPV4"}'
kubernetes.io/ingress.http-rules: "null"
kubernetes.io/ingress.https-rules: '[{"host":"example.tke.niewx.cn","path":"/","backend":{"serviceName":"springboot","servicePort":"8080"}}]'
kubernetes.io/ingress.rule-mix: "true"
nginx.ingress.kubernetes.io/permanent-redirect: https://www.baidu.com
nginx.ingress.kubernetes.io/permanent-redirect-code: "308"
nginx.ingress.kubernetes.io/use-regex: "true"我们访问域名测试下，可以发下最终后端是baidu，并且返回吗是308
[root@VM-0-3-centos ~]# curl -I -k https://example.tke.niewx.cn
HTTP/1.1 308 Permanent Redirect
Date: Wed, 30 Dec 2020 12:57:28 GMT
Content-Type: text/html
Content-Length: 164
Connection: keep-alive
Location: https://www.baidu.com
Strict-Transport-Security: max-age=15724800; includeSubDomains5. 客户端请求body的大小
我通过ingress给我的harbor配置一个域名，但是我上次镜像的时间报错了，报错413 Request Entity Too Large，这个是客户端上传镜像到仓库的请求body太大，需要怎么解决呢？其实ingress中提供了注解用来配置请求体的大小即可，加上注解nginx.ingress.kubernetes.io/proxy-body-size: 1000m，具体的大小限制根据自身需求配置即可。
apiVersion: networking.k8s.io/v1beta1
kind: Ingress
metadata:
annotations:
ingress.cloud.tencent.com/direct-access: "false"
kubernetes.io/ingress.class: nwx-ingress
kubernetes.io/ingress.extensiveParameters: '{"AddressIPVersion":"IPV4"}'
kubernetes.io/ingress.http-rules: '[{"host":"harbor.tke.niewx.cn","path":"/","backend":{"serviceName":"harbor-1610088954-harbor-portal","servicePort":"80"}}]'
kubernetes.io/ingress.https-rules: "null"
kubernetes.io/ingress.rule-mix: "false"
nginx.ingress.kubernetes.io/proxy-body-size: 1000m
nginx.ingress.kubernetes.io/use-regex: "true"
name: harbor-ingress
namespace: harbor6. http2-max-requests配置
问题现象：通过公网长连接访问后ingress后, 脚本每次请求1000次后,到1001次， grpc抛出: connections to all backends failing
github相关问题issue：https://github.com/kubernetes/ingress-nginx/issues/3028
解决方案：这里是因为nginx中http2-max-requests这个配置配置太小，建议的解决方法是参考https://trac.nginx.org/nginx/ticket/1250
链接的的建议When we set http2_max_requests 1000000, the possibility became very low.
这里在nginx-ingress需要设置下controller的全局configmap配置，在配置中加下如下配置即可
http2-max-requests: "1000000"配置好之后，重启下nginx-ingress controller的pod即可。
7. nginx-ingress获取真实客户端ip
很多业务场景需要获取到真实客户端ip，如果是通过nginx-ingress提供访问，那么在后端的pod内怎么样才能获取到真实客户端ip呢？下面我们以nginx服务为例，当我们通过ingress的域名访问nginx服务，怎么样才能获取真实客户端ip。
如果不配置的话，nginx服务是获取不到真实客户端ip的。现在我们做如下配置
- 修改nginx-ingress controller的入口service配置，将externalTrafficPolicy改成Local
- 修改nginx-ingress controller的configmap配置，加上如下配置
compute-full-forwarded-for: "true"
forwarded-for-header: "X-Forwarded-For"
use-forwarded-headers: "true"修改后，nginx-ingress controller会自动加载configmap配置，我们在访问看看
这时候可以发现日志的客户端ip已经是真实的客户端ip了。
8. nginx-ingress代理tcp和udp服务
具体配置参考文档 https://www.niewx.cn/2021/11/09/2021-11-09-Use-Nginx-Ingress-to-expose-TCP-and-udp-services/
9. nginx-ingress同时支持http和https
大家在使用nginx-ingress的时候，经常会给域名配置证书，让域名走https协议访问，但是有的时候我们希望域名既能http访问，也能https访问， 当你在ingress配置了证书后，会发现通过http访问域名默认会跳转到https。
这个时候大家会有一个疑惑，我没配置强制跳转，为什么会http强制跳转到https呢？并且nginx-ingress不像云厂商一样支持混合协议的配置，如果我希望域名能同时通过http和https访问到，该怎么配置呢？下面我来给大家讲解下
首先你在ingress配置了证书，没配置强制跳转，但是http直接强制到了https，这是因为Ingress 启用了 TLS，控制器会使用 308永久重定向响应将HTTP客户端重定向到HTTPS端口 443，Ingress 里配置了 https 证书的话，默认就一定会走 https。
那么我们要如何关闭这个强制跳转呢？其实ingress提供一个注解来进行控制
nginx.ingress.kubernetes.io/ssl-redirect: “false”如果你希望不进行强制跳转，那你在ingress加上这个注解即可。
从测试可以发现，加上注解后，http和https都可以进行访问了。
10. nginx-ingress实现金丝雀发布
参考tke官网文档配置即可 https://cloud.tencent.com/document/product/457/48907
11. nginx-ingress实现grpc转发
参考下面文档配置
https://cloud.tencent.com/developer/article/1970910
12. nginx-ingress如何开启tls 1.0和1.1
官网文档说明：https://kubernetes.github.io/ingress-nginx/user-guide/tls/
修改nginx ingress controller的配置文件，加上如下配置即可，tke控制台可以直接组件里面进行修改
ssl-ciphers: "ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-CHACHA20-POLY1305:ECDHE-RSA-CHACHA20-POLY1305:DHE-RSA-AES128-GCM-SHA256:DHE-RSA-AES256-GCM-SHA384:DHE-RSA-CHACHA20-POLY1305:ECDHE-ECDSA-AES128-SHA256:ECDHE-RSA-AES128-SHA256:ECDHE-ECDSA-AES128-SHA:ECDHE-RSA-AES128-SHA:ECDHE-ECDSA-AES256-SHA384:ECDHE-RSA-AES256-SHA384:ECDHE-ECDSA-AES256-SHA:ECDHE-RSA-AES256-SHA:DHE-RSA-AES128-SHA256:DHE-RSA-AES256-SHA256:AES128-GCM-SHA256:AES256-GCM-SHA384:AES128-SHA256:AES256-SHA256:AES128-SHA:AES256-SHA:DES-CBC3-SHA"
ssl-protocols: "TLSv1 TLSv1.1 TLSv1.2 TLSv1.3"13. nginx-ingress重试机制
问题现象：通过ingress的域名访问后端，调用一次，服务日志里显示调用了三次，nginx-ingress的请求日志出现了3次返回码
- - [2022-09-14T11:40:27+00:00] 1663155627.705 "GET
/api HTTP/1.1" 502 150 "-" "curl/7.29.0" 121
3.073 [background-sino-xray-9091] [] [10.160.253.235:9091,
10.160.253.235:9091, 10.160.253.235:9091] [0, 0, 0] [1.028, 1.024,
1.024] [502, 502, 502] 7d407c28f5d7d2aae3afd0c368fbe9e5问题原因：从访问日志看，nginx ingress controller出现3次调后端rs，为什么会调用了3次呢？这里是因为nginx本身的重试机制，具体参数可以参考文档
http://nginx.org/en/docs/http/ngx_http_proxy_module.html#proxy_next_upstream
但是为什么是重试3次呢，这里可以到nginx-ingress controller看下配置文件
proxy_next_upstream error timeout; #哪些情况下应将请求传递到下一台服务器
proxy_next_upstream_timeout 0; # 限制可以将请求传递到下一台服务器的时间，0表示将关闭此限制
proxy_next_upstream_tries 3; # 限制将请求传递到下一台服务器的可能尝试次数。0表示关闭此限制可以看出tke的nginx ingress controller默认是重试3次，因此日志里会显示调用了3次。如果要关闭重试，直接在全局configmap里proxy_next_upstream_tries设置为0
14. nginx-ingress无法reload全局配置
https://cloud.tencent.com/developer/article/2040556
15. nginx-ingress超时配置
nginx ingress的upstream超时设置主要是设置下面3个参数，单位都是秒(s)
proxy_connect_timeout upstream 后端连接超时时间。
proxy_read_timeout 读取 upstream 后端响应超时时间。
proxy_send_timeout 向 upstream 后端发送请求的超时时间。如果是全局配置可以修改nginx ingress controller的全局configmap
apiVersion: v1
kind: ConfigMap
metadata:
name: nginx-configuration
namespace: ingress-nginx
labels:
app.kubernetes.io/instance: ingress-nginx
app.kubernetes.io/part-of: ingress-nginx
data:
proxy-connect-timeout: "30"
proxy-send-timeout: "60"
proxy-read-timeout: "60"如果是给单个ingress设置，可以给ingress加上注解
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
name: sample-ingress
annotations:
nginx.ingress.kubernetes.io/proxy-connect-timeout: "600"
nginx.ingress.kubernetes.io/proxy-send-timeout: "600"
nginx.ingress.kubernetes.io/proxy-read-timeout: "600"
spec:
rules:
- host: example.com
http:
paths:
- path: /
pathType: Prefix
backend:
service:
name: sample-service
port:
number: 80配置注解的时候需要注意下，不能加上单位，否则会配置不生效，比如600s，直接配置为600，这几个注解vaule的类型都是number类型，只能是数字。
https://kubernetes.github.io/ingress-nginx/user-guide/nginx-configuration/annotations/
腾讯云开发者
