Knative健康检查

李鹤

发布于 2023-03-06 11:19:02

4030

发布于 2023-03-06 11:19:02

文章被收录于专栏：k-cloud-labs

背景

knative 0.14.0 实际修改可能与贴出来的代码不符，贴出来的代码只是为了方便快速实现功能

在支持了前面的定制功能后，集群中部署ksvc服务时会报IngressNotConfigured错误

原因分析

首先根据错误提示及日志信息，可以发现是在做健康检查的时候出的问题，期望得到200，但是得到了404

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35

func (m *Prober) probeVerifier(item *workItem) prober.Verifier { return func(r *http.Response, _ []byte) (bool, error) { // In the happy path, the probe request is forwarded to Activator or Queue-Proxy and the response (HTTP 200) // contains the "K-Network-Hash" header that can be compared with the expected hash. If the hashes match, // probing is successful, if they don't match, a new probe will be sent later. // An HTTP 404/503 is expected in the case of the creation of a new Knative service because the rules will // not be present in the Envoy config until the new VirtualService is applied. // No information can be extracted from any other scenario (e.g. HTTP 302), therefore in that case, // probing is assumed to be successful because it is better to say that an Ingress is Ready before it // actually is Ready than never marking it as Ready. It is best effort. switch r.StatusCode { case http.StatusOK: hash := r.Header.Get(network.HashHeaderName) switch hash { case "": m.logger.Errorf("Probing of %s abandoned, IP: %s:%s: the response doesn't contain the %q header", item.url, item.podIP, item.podPort, network.HashHeaderName) return true, nil case item.ingressState.hash: return true, nil default: m.logger.Warnf("unexpected hash: want %q, got %q", item.ingressState.hash, hash) return true, nil } // 日志中报错的地方，探活希望得到200，但是得到了404 case http.StatusNotFound, http.StatusServiceUnavailable: return false, fmt.Errorf("unexpected status code: want %v, got %v", http.StatusOK, http.StatusNotFound) default: m.logger.Errorf("Probing of %s abandoned, IP: %s:%s: the response status is %v, expected 200 or 404", item.url, item.podIP, item.podPort, r.StatusCode) return true, nil } } }

其实这时候大致也能猜到是什么原因了，因为我们定制了通过USN进行过滤，探活的时候，Url中其实是没有USN的。下一步就是顺藤摸瓜，找到探活对应的代码验证我们的猜想，也比较简单

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19

// processWorkItem processes a single work item from workQueue. // It returns false when there is no more items to process, true otherwise. func (m *Prober) processWorkItem() bool { ... // probePath /healthz probeURL.Path = path.Join(probeURL.Path, probePath) ok, err := prober.Do( item.context, transport, probeURL.String(), prober.WithHeader(network.UserAgentKey, network.IngressReadinessUserAgent), prober.WithHeader(network.ProbeHeaderName, network.ProbeHeaderValue), m.probeVerifier(item)) ... }

可以看到探活的时候就是拿Path拼上/heathz，验证了我们的猜想

修复

修改也就比较简单了，在添加wotkitem时，预先把USN添加到path中即可

1 2 3 4 5 6 7 8 9 10 11 12 13 14

func (l *gatewayPodTargetLister) ListProbeTargets(ctx context.Context, ing *v1alpha1.Ingress) ([]status.ProbeTarget, error) { ... // Use sorted hosts list for consistent ordering. for i, host := range gatewayHosts[gatewayName].List() { newURL := *target.URLs[0] newURL.Host = host + ":" + target.Port var usn string if ing.Annotations != nil { usn = ing.Annotations["serverless.kakuchuxing.com/usn"] } newURL.Path = path.Join(newURL.Path, usn) qualifiedTarget.URLs[i] = &newURL ... }