作者杜杨浩,腾讯云高级工程师,热衷于开源、容器和Kubernetes。目前主要从事Kubernetes集群高可用&备份还原,以及边缘计算相关研发工作
整个Kubernetes技术体系由声明式API以及Controller构成,而kube-apiserver是Kubernetes的声明式api server,并为其它组件交互提供了桥梁。因此加深对kube-apiserver的理解就显得至关重要了。
kube-apiserver作为整个Kubernetes集群操作etcd的唯一入口,负责Kubernetes各资源的认证&鉴权,校验以及CRUD等操作,提供RESTful APIs,供其它组件调用:
kube-apiserver包含三种APIServer:
apiregistration.k8s.io
组下的APIService资源请求,同时将来自用户的请求拦截转发给aggregated server(AA)另外还包括bootstrap-controller,主要负责Kubernetes default apiserver service的创建以及管理。
接下来将对上述组件进行概览性总结。
--service-cluster-ip-range
指定范围)--service-node-port-range
指定范围)// k8s.io/kubernetes/pkg/master/controller.go:142
// Start begins the core controller loops that must exist for bootstrapping
// a cluster.
func (c *Controller) Start() {
if c.runner != nil {
return
}
// Reconcile during first run removing itself until server is ready.
endpointPorts := createEndpointPortSpec(c.PublicServicePort, "https", c.ExtraEndpointPorts)
if err := c.EndpointReconciler.RemoveEndpoints(kubernetesServiceName, c.PublicIP, endpointPorts); err != nil {
klog.Errorf("Unable to remove old endpoints from kubernetes service: %v", err)
}
repairClusterIPs := servicecontroller.NewRepair(c.ServiceClusterIPInterval, c.ServiceClient, c.EventClient, &c.ServiceClusterIPRange, c.ServiceClusterIPRegistry, &c.SecondaryServiceClusterIPRange, c.SecondaryServiceClusterIPRegistry)
repairNodePorts := portallocatorcontroller.NewRepair(c.ServiceNodePortInterval, c.ServiceClient, c.EventClient, c.ServiceNodePortRange, c.ServiceNodePortRegistry)
// run all of the controllers once prior to returning from Start.
if err := repairClusterIPs.RunOnce(); err != nil {
// If we fail to repair cluster IPs apiserver is useless. We should restart and retry.
klog.Fatalf("Unable to perform initial IP allocation check: %v", err)
}
if err := repairNodePorts.RunOnce(); err != nil {
// If we fail to repair node ports apiserver is useless. We should restart and retry.
klog.Fatalf("Unable to perform initial service nodePort check: %v", err)
}
// 定期执行bootstrap controller主要的四个功能(reconciliation)
c.runner = async.NewRunner(c.RunKubernetesNamespaces, c.RunKubernetesService, repairClusterIPs.RunUntil, repairNodePorts.RunUntil)
c.runner.Start()
}
更多代码原理详情,参考 kubernetes-reading-notes[1] 。
KubeAPIServer主要提供对内建API Resources的操作请求,为Kubernetes中各API Resources注册路由信息,同时暴露RESTful API,使集群中以及集群外的服务都可以通过RESTful API操作Kubernetes中的资源
另外,kubeAPIServer是整个Kubernetes apiserver的核心,下面将要讲述的aggregatorServer以及apiExtensionsServer都是建立在kubeAPIServer基础上进行扩展的(补充了Kubernetes对用户自定义资源的能力支持)
kubeAPIServer最核心的功能是为Kubernetes内置资源添加路由,如下:
m.InstallLegacyAPI
将核心 API Resources添加到路由中,在apiserver中即是以 /api
开头的 resource;m.InstallAPIs
将扩展的 API Resources添加到路由中,在apiserver中即是以 /apis
开头的 resource;// k8s.io/kubernetes/pkg/master/master.go:332
// New returns a new instance of Master from the given config.
// Certain config fields will be set to a default value if unset.
// Certain config fields must be specified, including:
// KubeletClientConfig
func (c completedConfig) New(delegationTarget genericapiserver.DelegationTarget) (*Master, error) {
...
// 安装 LegacyAPI(core API)
// install legacy rest storage
if c.ExtraConfig.APIResourceConfigSource.VersionEnabled(apiv1.SchemeGroupVersion) {
legacyRESTStorageProvider := corerest.LegacyRESTStorageProvider{
StorageFactory: c.ExtraConfig.StorageFactory,
ProxyTransport: c.ExtraConfig.ProxyTransport,
KubeletClientConfig: c.ExtraConfig.KubeletClientConfig,
EventTTL: c.ExtraConfig.EventTTL,
ServiceIPRange: c.ExtraConfig.ServiceIPRange,
SecondaryServiceIPRange: c.ExtraConfig.SecondaryServiceIPRange,
ServiceNodePortRange: c.ExtraConfig.ServiceNodePortRange,
LoopbackClientConfig: c.GenericConfig.LoopbackClientConfig,
ServiceAccountIssuer: c.ExtraConfig.ServiceAccountIssuer,
ServiceAccountMaxExpiration: c.ExtraConfig.ServiceAccountMaxExpiration,
APIAudiences: c.GenericConfig.Authentication.APIAudiences,
}
if err := m.InstallLegacyAPI(&c, c.GenericConfig.RESTOptionsGetter, legacyRESTStorageProvider); err != nil {
return nil, err
}
}
...
// 安装 APIs(named groups apis)
if err := m.InstallAPIs(c.ExtraConfig.APIResourceConfigSource, c.GenericConfig.RESTOptionsGetter, restStorageProviders...); err != nil {
return nil, err
}
...
return m, nil
}
整个kubeAPIServer提供了三类API Resource接口:
/api/v1
下;/apis/$GROUP/$VERSION
;/metrics
、/version
等;而API的URL大致以 /apis/{group}/{version}/namespaces/{namespace}/resource/{name}
组成,结构如下图所示:
kubeAPIServer会为每种API资源创建对应的RESTStorage,RESTStorage的目的是将每种资源的访问路径及其后端存储的操作对应起来:通过构造的REST Storage实现的接口判断该资源可以执行哪些操作(如:create、update等),将其对应的操作存入到action中,每一个操作对应一个标准的REST method,如create对应REST method为POST,而update对应REST method为PUT。最终根据actions数组依次遍历,对每一个操作添加一个handler(handler对应REST Storage实现的相关接口),并注册到route,最终对外提供RESTful API,如下:
// m.GenericAPIServer.InstallLegacyAPIGroup --> s.installAPIResources --> apiGroupVersion.InstallREST --> installer.Install --> a.registerResourceHandlers
// k8s.io/kubernetes/staging/src/k8s.io/apiserver/pkg/endpoints/installer.go:181
func (a *APIInstaller) registerResourceHandlers(path string, storage rest.Storage, ws *restful.WebService) (*metav1.APIResource, error) {
...
// 1、判断该 resource 实现了哪些 REST 操作接口,以此来判断其支持的 verbs 以便为其添加路由
// what verbs are supported by the storage, used to know what verbs we support per path
creater, isCreater := storage.(rest.Creater)
namedCreater, isNamedCreater := storage.(rest.NamedCreater)
lister, isLister := storage.(rest.Lister)
getter, isGetter := storage.(rest.Getter)
...
// 2、为 resource 添加对应的 actions(+根据是否支持 namespace)
// Get the list of actions for the given scope.
switch {
case !namespaceScoped:
// Handle non-namespace scoped resources like nodes.
resourcePath := resource
resourceParams := params
itemPath := resourcePath + "/{name}"
nameParams := append(params, nameParam)
proxyParams := append(nameParams, pathParam)
...
// Handler for standard REST verbs (GET, PUT, POST and DELETE).
// Add actions at the resource path: /api/apiVersion/resource
actions = appendIf(actions, action{"LIST", resourcePath, resourceParams, namer, false}, isLister)
actions = appendIf(actions, action{"POST", resourcePath, resourceParams, namer, false}, isCreater)
...
}
...
// 3、从 rest.Storage 到 restful.Route 映射
// 为每个操作添加对应的 handler
for _, action := range actions {
...
switch action.Verb {
...
case "POST": // Create a resource.
var handler restful.RouteFunction
// 4、初始化 handler
if isNamedCreater {
handler = restfulCreateNamedResource(namedCreater, reqScope, admit)
} else {
handler = restfulCreateResource(creater, reqScope, admit)
}
handler = metrics.InstrumentRouteFunc(action.Verb, group, version, resource, subresource, requestScope, metrics.APIServerComponent, handler)
...
// 5、route 与 handler 进行绑定
route := ws.POST(action.Path).To(handler).
Doc(doc).
Param(ws.QueryParameter("pretty", "If 'true', then the output is pretty printed.")).
Operation("create"+namespaced+kind+strings.Title(subresource)+operationSuffix).
Produces(append(storageMeta.ProducesMIMETypes(action.Verb), mediaTypes...)...).
Returns(http.StatusOK, "OK", producedObject).
// TODO: in some cases, the API may return a v1.Status instead of the versioned object
// but currently go-restful can't handle multiple different objects being returned.
Returns(http.StatusCreated, "Created", producedObject).
Returns(http.StatusAccepted, "Accepted", producedObject).
Reads(defaultVersionedObject).
Writes(producedObject)
if err := AddObjectParams(ws, route, versionedCreateOptions); err != nil {
return nil, err
}
addParams(route, action.Params)
// 6、添加到路由中
routes = append(routes, route)
case "DELETE": // Delete a resource.
...
default:
return nil, fmt.Errorf("unrecognized action verb: %s", action.Verb)
}
for _, route := range routes {
route.Metadata(ROUTE_META_GVK, metav1.GroupVersionKind{
Group: reqScope.Kind.Group,
Version: reqScope.Kind.Version,
Kind: reqScope.Kind.Kind,
})
route.Metadata(ROUTE_META_ACTION, strings.ToLower(action.Verb))
ws.Route(route)
}
// Note: update GetAuthorizerAttributes() when adding a custom handler.
}
...
}
kubeAPIServer代码结构整理如下:
1. apiserver整体启动逻辑 k8s.io/kubernetes/cmd/kube-apiserver
2. apiserver bootstrap-controller创建&运行逻辑 k8s.io/kubernetes/pkg/master
3. API Resource对应后端RESTStorage(based on genericregistry.Store)创建 k8s.io/kubernetes/pkg/registry
4. aggregated-apiserver创建&处理逻辑 k8s.io/kubernetes/staging/src/k8s.io/kube-aggregator
5. extensions-apiserver创建&处理逻辑 k8s.io/kubernetes/staging/src/k8s.io/apiextensions-apiserver
6. apiserver创建&运行 k8s.io/kubernetes/staging/src/k8s.io/apiserver/pkg/server
7. 注册API Resource资源处理handler(InstallREST&Install®isterResourceHandlers) k8s.io/kubernetes/staging/src/k8s.io/apiserver/pkg/endpoints
8. 创建存储后端(etcdv3) k8s.io/kubernetes/staging/src/k8s.io/apiserver/pkg/storage
genericregistry.Store.CompleteWithOptions初始化 k8s.io/kubernetes/staging/src/k8s.io/apiserver/pkg/registry
调用链整理如下:
更多代码原理详情,参考 kubernetes-reading-notes[1] 。
aggregatorServer主要用于处理扩展Kubernetes API Resources的第二种方式Aggregated APIServer(AA),将CR请求代理给AA:
这里结合Kubernetes官方给出的aggregated apiserver例子sample-apiserver,总结原理如下:
GenericAPIServer
和维护自身状态的Controller
。其中GenericAPIServer
主要处理apiregistration.k8s.io
组下的APIService资源请求,而Controller包括:apiserviceRegistrationController
:负责根据APIService定义的aggregated server service构建代理,将CR的请求转发给后端的aggregated serveravailableConditionController
:维护 APIServices 的可用状态,包括其引用 Service 是否可用等;autoRegistrationController
:用于保持 API 中存在的一组特定的 APIServices;crdRegistrationController
:负责将 CRD GroupVersions 自动注册到 APIServices 中;openAPIAggregationController
:将 APIServices 资源的变化同步至提供的 OpenAPI 文档;$VERSION.$GROUP
,这些APIService都会有标签kube-aggregator.kubernetes.io/automanaged: onstart
,例如:v1.apps apiService。autoRegistrationController创建并维护这些列表中的APIService,也即我们看到的Local apiService;对于自定义的APIService(aggregated server),则不会对其进行处理更多代码原理详情,参考 kubernetes-reading-notes[1] 。
apiExtensionsServer主要负责CustomResourceDefinition(CRD)apiResources以及apiVersions的注册,同时处理CRD以及相应CustomResource(CR)的REST请求(如果对应CR不能被处理的话则会返回404),也是apiserver Delegation的最后一环
原理总结如下:
crdRegistrationController
负责将CRD GroupVersions自动注册到APIServices中。具体逻辑为:枚举所有CRDs,然后根据CRD定义的crd.Spec.Group以及crd.Spec.Versions字段构建APIService,并添加到autoRegisterController.apiServicesToSync中,由autoRegisterController进行创建以及维护操作。这也是为什么创建完CRD后会产生对应的APIService对象openapiController
:将 crd 资源的变化同步至提供的 OpenAPI 文档,可通过访问 /openapi/v2
进行查看;crdController
:负责将 crd 信息注册到 apiVersions 和 apiResources 中,两者的信息可通过 kubectl api-versions
和 kubectl api-resources
查看;kubectl api-versions
命令返回所有Kubernetes集群资源的版本信息(实际发出了两个请求,分别是https://127.0.0.1:6443/api
以及https://127.0.0.1:6443/apis
,并在最后将两个请求的返回结果进行了合并)
$ kubectl -v=8 api-versions I1211 11:44:50.276446 22493 loader.go:375] Config loaded from file: /root/.kube/config I1211 11:44:50.277005 22493 round_trippers.go:420] GET https://127.0.0.1:6443/api?timeout=32s ... I1211 11:44:50.290265 22493 request.go:1068] Response Body: {"kind":"APIVersions","versions":["v1"],"serverAddressByClientCIDRs":[{"clientCIDR":"0.0.0.0/0","serverAddress":"x.x.x.x:6443"}]} I1211 11:44:50.293673 22493 round_trippers.go:420] GET https://127.0.0.1:6443/apis?timeout=32s ... I1211 11:44:50.298360 22493 request.go:1068] Response Body: {"kind":"APIGroupList","apiVersion":"v1","groups":[{"name":"apiregistration.k8s.io","versions":[{"groupVersion":"apiregistration.k8s.io/v1","version":"v1"},{"groupVersion":"apiregistration.k8s.io/v1beta1","version":"v1beta1"}],"preferredVersion":{"groupVersion":"apiregistration.k8s.io/v1","version":"v1"}},{"name":"extensions","versions":[{"groupVersion":"extensions/v1beta1","version":"v1beta1"}],"preferredVersion":{"groupVersion":"extensions/v1beta1","version":"v1beta1"}},{"name":"apps","versions":[{"groupVersion":"apps/v1","version":"v1"}],"preferredVersion":{"groupVersion":"apps/v1","version":"v1"}},{"name":"events.k8s.io","versions":[{"groupVersion":"events.k8s.io/v1beta1","version":"v1beta1"}],"preferredVersion":{"groupVersion":"events.k8s.io/v1beta1","version":"v1beta1"}},{"name":"authentication.k8s.io","versions":[{"groupVersion":"authentication.k8s.io/v1","version":"v1"},{"groupVersion":"authentication.k8s.io/v1beta1","version":"v1beta1"}],"preferredVersion":{"groupVersion":"authentication.k8s.io/v1"," [truncated 4985 chars] apiextensions.k8s.io/v1 apiextensions.k8s.io/v1beta1 apiregistration.k8s.io/v1 apiregistration.k8s.io/v1beta1 apps/v1 authentication.k8s.io/v1beta1 ... storage.k8s.io/v1 storage.k8s.io/v1beta1 v1kubectl api-resources
命令就是先获取所有API版本信息,然后对每一个API版本调用接口获取该版本下的所有API资源类型
$ kubectl -v=8 api-resources 5077 loader.go:375] Config loaded from file: /root/.kube/config I1211 15:19:47.593450 15077 round_trippers.go:420] GET https://127.0.0.1:6443/api?timeout=32s I1211 15:19:47.602273 15077 request.go:1068] Response Body: {"kind":"APIVersions","versions":["v1"],"serverAddressByClientCIDRs":[{"clientCIDR":"0.0.0.0/0","serverAddress":"x.x.x.x:6443"}]} I1211 15:19:47.606279 15077 round_trippers.go:420] GET https://127.0.0.1:6443/apis?timeout=32s I1211 15:19:47.610333 15077 request.go:1068] Response Body: {"kind":"APIGroupList","apiVersion":"v1","groups":[{"name":"apiregistration.k8s.io","versions":[{"groupVersion":"apiregistration.k8s.io/v1","version":"v1"},{"groupVersion":"apiregistration.k8s.io/v1beta1","version":"v1beta1"}],"preferredVersion":{"groupVersion":"apiregistration.k8s.io/v1","version":"v1"}},{"name":"extensions","versions":[{"groupVersion":"extensions/v1beta1","version":"v1beta1"}],"preferredVersion":{"groupVersion":"extensions/v1beta1","version":"v1beta1"}},{"name":"apps","versions":[{"groupVersion":"apps/v1","version":"v1"}],"preferredVersion":{"groupVersion":"apps/v1","version":"v1"}},{"name":"events.k8s.io","versions":[{"groupVersion":"events.k8s.io/v1beta1","version":"v1beta1"}],"preferredVersion":{"groupVersion":"events.k8s.io/v1beta1","version":"v1beta1"}},{"name":"authentication.k8s.io","versions":[{"groupVersion":"authentication.k8s.io/v1","version":"v1"},{"groupVersion":"authentication.k8s.io/v1beta1","version":"v1beta1"}],"preferredVersion":{"groupVersion":"authentication.k8s.io/v1"," [truncated 4985 chars] I1211 15:19:47.614700 15077 round_trippers.go:420] GET https://127.0.0.1:6443/apis/batch/v1?timeout=32s I1211 15:19:47.614804 15077 round_trippers.go:420] GET https://127.0.0.1:6443/apis/authentication.k8s.io/v1?timeout=32s I1211 15:19:47.615687 15077 round_trippers.go:420] GET https://127.0.0.1:6443/apis/auth.tkestack.io/v1?timeout=32s https://127.0.0.1:6443/apis/authentication.k8s.io/v1beta1?timeout=32s I1211 15:19:47.616794 15077 round_trippers.go:420] GET https://127.0.0.1:6443/apis/coordination.k8s.io/v1?timeout=32s I1211 15:19:47.616863 15077 round_trippers.go:420] GET https://127.0.0.1:6443/apis/apps/v1?timeout=32s ... NAME SHORTNAMES APIGROUP NAMESPACED KIND bindings true Binding endpoints ep true Endpoints events ev true Event limitranges limits true LimitRange namespaces ns false Namespace nodes no false Node ...namingController
:检查 crd obj 中是否有命名冲突,可在 crd .status.conditions
中查看;establishingController
:检查 crd 是否处于正常状态,可在 crd .status.conditions
中查看;nonStructuralSchemaController
:检查 crd obj 结构是否正常,可在 crd .status.conditions
中查看;apiApprovalController
:检查 crd 是否遵循 Kubernetes API 声明策略,可在 crd .status.conditions
中查看;finalizingController
:类似于 finalizes 的功能,与 CRs 的删除有关;总结CR CRUD APIServer处理逻辑如下:
更多代码原理详情,参考 kubernetes-reading-notes[1] 。
本文从源码层面对Kubernetes apiserver进行了一个概览性总结,包括:aggregatorServer,kubeAPIServer,apiExtensionsServer以及bootstrap-controller等。通过阅读本文可以对apiserver内部原理有一个大致的理解,另外也有助于后续深入研究。
[1]
kubernetes-reading-notes: https://github.com/duyanghao/kubernetes-reading-notes/tree/master/core/api-server
回答问题拿礼品
公众号留言板评论以下【问题编号+答案】
全部问题回答正确的同学
将有机会获得腾讯云公仔1只 仅限三个名额,快快参与起来吧~
问题1:如果手动将APIService删掉了,会自动恢复吗?如果删Aggregated APIserver对应APIService呢? 问题2:从代码角度看,你觉得kube-apiserver的设计怎么样?
问题3:kube-apiserver从接受创建请求到完成资源的创建,会经历decode,admission,validation以及encode的流程,你知道它们分别是什么作用吗?
活动截止时间:2021年1月18日18:00
插播福利!!!
腾讯云原生后台回复关键字“手册” 即可获取
《腾讯云原生路线图手册》&《腾讯云原生最佳实践》