使用Aggregated APIServer的方式构建API服务



尽管可以使用gin, go-restful等go语言web框架轻易地构建出一个稳定的API接口服务,但以kubernetes原生的方式构建API接口服务还是有很多吸引人的好处的。官方文档中已经将这些好处列出了:

User-Defined Types Consider adding a Custom Resource to Kubernetes if you want to define new controllers, application configuration objects or other declarative APIs, and to manage them using Kubernetes tools, such as kubectl. Do not use a Custom Resource as data storage for application, user, or monitoring data. For more about Custom Resources, see the Custom Resources concept guide. Combining New APIs with Automation The combination of a custom resource API and a control loop is called the Operator pattern. The Operator pattern is used to manage specific, usually stateful, applications. These custom APIs and control loops can also be used to control other resources, such as storage or policies. Changing Built-in Resources When you extend the Kubernetes API by adding custom resources, the added resources always fall into a new API Groups. You cannot replace or change existing API groups. Adding an API does not directly let you affect the behavior of existing APIs (e.g. Pods), but API Access Extensions do. API Access Extensions When a request reaches the Kubernetes API Server, it is first Authenticated, then Authorized, then subject to various types of Admission Control. See Controlling Access to the Kubernetes API for more on this flow. Each of these steps offers extension points. Kubernetes has several built-in authentication methods that it supports. It can also sit behind an authenticating proxy, and it can send a token from an Authorization header to a remote service for verification (a webhook). All of these methods are covered in the Authentication documentation. Authentication Authentication maps headers or certificates in all requests to a username for the client making the request. Kubernetes provides several built-in authentication methods, and an Authentication webhook method if those don’t meet your needs. Authorization Authorization determines whether specific users can read, write, and do other operations on API resources. It just works at the level of whole resources – it doesn’t discriminate based on arbitrary object fields. If the built-in authorization options don’t meet your needs, and Authorization webhook allows calling out to user-provided code to make an authorization decision. Dynamic Admission Control After a request is authorized, if it is a write operation, it also goes through Admission Control steps. In addition to the built-in steps, there are several extensions:

  • The Image Policy webhook restricts what images can be run in containers.
  • To make arbitrary admission control decisions, a general Admission webhook can be used. Admission Webhooks can reject creations or updates.



官方提供了两种方式以实现对标准kubernetes API接口的扩展:1)Aggregated APIServer 2)Custom Resource

两种方式的区别是定义api-resource的方式不同。在Aggregated APIServer方式中,api-resource是通过代码向kubernetes注册资源类型的方式实现的,而Custom Resource是直接通过yaml文件创建自定义资源的方式实现的。


API Access Control Authentication

  • CR: All strategies supported. Configured by root apiserver.
  • AA: Supporting all root apiserver’s authenticating strategies but it has to be done via authentication token review apiexcept for authentication proxy which will cause an extra cost of network RTT.


  • CR: All strategies supported. Configured by root apiserver.
  • AA: Delegating authorization requests to root apiserver via SubjectAccessReview api. Note that this approach will also cost a network RTT.

Admission Control

  • CR: You could extend via dynamic admission control webhook (which is costing network RTT).
  • AA: While You can develop and customize your own admission controller which is dedicated to your AA. While You can’t reuse root-apiserver’s built-in admission controllers nomore.

API Schema Note: CR’s integration with OpenAPI schema is being enhanced in the future releases and it will have a stronger integration with OpenAPI mechanism. Validating

  • CR: (landed in 1.12) Defined via OpenAPIv3 Schema grammar. more
  • AA: You can customize any validating flow you want.


  • CR: (landed in 1.13) The CR conversioning (basically from storage version to requested version) could be done via conversioning webhook.
  • AA: Develop any conversion you want.


  • CR: Currently only status and scale sub-resource supported.
  • AA: You can customize any sub-resouce you want.

OpenAPI Schema

  • CR: (landed in 1.13) The corresponding CRD’s OpenAPI schema will be automatically synced to root-apiserver’s openapi doc api.
  • AA: OpenAPI doc has to be manually generated by code-generating tools.

Other Functionalities AA (Aggregated APIServer) CR (Custom Resource) SMP(Strategic Merge Patch) Supported Not yet. Will be replaced via server-side apply instead Informative Kubectl Printing Not supported, unless you develop your own with server-side printing. By AdditionalPrinterColumns Websocket/(Other non-HTTP transport) Supported No metadata.GenerationAuto Increment Supported Nope, and this is designed Use Another Backend/Secondary Storage Supported For now, ETCD3 only More Comparision here




虽然官方给了一个sample-apiserver,我们可以照着实现自己的Aggregated APIServer。但完全手工编写还是太费劲了,这里使用官方推荐的工具apiserver-builder帮助快速创建项目骨架。




# 创建项目目录
mkdir $GOPATH/src/github.com/jeremyxu2010/demo-apiserver
# 在项目目录下新建一个名为boilerplate.go.txt,里面是代码的头部版权声明
cd $GOPATH/src/github.com/jeremyxu2010/demo-apiserver
curl -o boilerplate.go.txt https://github.com/kubernetes/kubernetes/blob/master/hack/boilerplate/boilerplate.go.txt
# 初始化项目
apiserver-boot init repo --domain jeremyxu2010.me
# 创建一个非命名空间范围的api-resource
apiserver-boot create group version resource --group demo --version v1beta1 --non-namespaced=true --kind Foo
# 创建Foo这个api-resource的子资源
apiserver-boot create subresource --subresource bar --group demo --version v1beta1 --kind Foo
# 生成上述创建的api-resource类型的相关代码,包括deepcopy接口实现代码、versioned/unversioned类型转换代码、api-resource类型注册代码、api-resource类型的Controller代码、api-resource类型的AdmissionController代码
apiserver-boot build generated
# 直接在本地将etcd, apiserver, controller运行起来
apiserver-boot run local


curl -k


# 创建Foo资源的yaml
echo 'apiVersion: demo.jeremyxu2010.me/v1beta1
kind: Foo
  name: foo-example
  namespace: test
spec: {}' > sample/foo.yaml

# 查看已经注册的api-resource类型
kubectl --kubeconfig api-resources
# 列所有foos
kubectl --kubeconfig kubeconfig get foos
# 创建foo
kubectl --kubeconfig kubeconfig create -f sample/foo.yaml
# 再列所有foos
kubectl --kubeconfig kubeconfig get foos
# Get新创建的foo
kubectl --kubeconfig kubeconfig get foos foo-example
kubectl --kubeconfig kubeconfig get foos foo-example -o yaml
# Delete新创建的foo
kubectl --kubeconfig kubeconfig delete foos foo-example


func main() {
	version := "v0"
	server.StartApiServer("/registry/jeremyxu2010.me", apis.GetAllApiBuilders(), openapi.GetOpenAPIDefinitions, "Api", version, func(apiServerConfig *apiserver.Config) error {
		apiServerConfig.RecommendedConfig.EnableSwaggerUI = true
		apiServerConfig.RecommendedConfig.SwaggerConfig = genericapiserver.DefaultSwaggerConfig()
		return nil




type Foo struct {
	metav1.TypeMeta   `json:",inline"`
	metav1.ObjectMeta `json:"metadata,omitempty"`

	Spec   FooSpec   `json:"spec,omitempty"`
	Status FooStatus `json:"status,omitempty"`

// FooSpec defines the desired state of Foo
type FooSpec struct {

// FooStatus defines the observed state of Foo
type FooStatus struct {






// Reconcile reads that state of the cluster for a Foo object and makes changes based on the state read
// and what is in the Foo.Spec
// TODO(user): Modify this Reconcile function to implement your Controller logic.  The scaffolding writes
// a Deployment as an example
// +kubebuilder:rbac:groups=demo.jeremyxu2010.me,resources=foos,verbs=get;list;watch;create;update;patch;delete
// +kubebuilder:rbac:groups=demo.jeremyxu2010.me,resources=foos/status,verbs=get;update;patch
func (r *ReconcileFoo) Reconcile(request reconcile.Request) (reconcile.Result, error) {
	// Fetch the Foo instance
	instance := &demov1beta1.Foo{}
	err := r.Get(context.TODO(), request.NamespacedName, instance)
	if err != nil {
		if errors.IsNotFound(err) {
			// Object not found, return.  Created objects are automatically garbage collected.
			// For additional cleanup logic use finalizers.
			return reconcile.Result{}, nil
		// Error reading the object - requeue the request.
		return reconcile.Result{}, err

	return reconcile.Result{}, nil


api-resource的admission controller编写可参考这里



# 生成二进制文件
apiserver-boot build executables
# 生成容器镜像
apiserver-boot build container --image demo/foo-apiserver:latest
# 生成kubernetes的部署manifest文件,可直接在kubernetes里apply即完成部署
apiserver-boot build config --name fool-apiserver --namespace default --image demo/foo-apiserver:latest




curl -o docs/openapi-spec/swagger.json
apiserver-build build docs --build-openapi=false --operations=true




apiserver-builder在生成代码时使用了一些kubernetes项目本身使用的code generator,这些code generator也挺有趣的,有时间可以仔细研究下。


编写Aggregated APIServer风格的API接口服务这一工作,终于接触到了kubernetes里的一些内部设计,不得不说这套设计还是相当简洁稳定的,难怪kubernetes项目最终能成功。




0 条评论
登录 后参与评论