Extend your kubernetes API with aggregated apiserver

Overview

What is Kubernetes aggregation

Kubernetes apiserver aggregationAA yes Kubernetes Provides an extension API Methods , There is no such thing as GA

Difference between CRD and AA

as everyone knows ,kubernetes Expand API There are about three ways to ：CRD、AA、 Manually extend the source code . according to CNCF Share in Min Kim Yes AA More focus on practice , Users do not need to understand the underlying principles , Used here kubebuilder, code-generator Our users can well understand this . The official also gave CRD And AA The difference between

API Access Control

Authentication

• CR: All strategies supported. Configured by root apiserver.
• AA: Supporting all root apiserver’s authenticating strategies but it has to be done via authentication token review api except for authentication proxy which will cause an extra cost of network RTT.

Authorization

• CR: All strategies supported. Configured by root apiserver.
• AA: Delegating authorization requests to root apiserver via SubjectAccessReview api. Note that this approach will also cost a network RTT.

Admission Control

• CR: You could extend via dynamic admission control webhook (which is costing network RTT).
• AA: While You can develop and customize your own admission controller which is dedicated to your AA. While You can’t reuse root-apiserver’s built-in admission controllers nomore.

API Schema

Note: CR’s integration with OpenAPI schema is being enhanced in the future releases and it will have a stronger integration with OpenAPI mechanism.

Validating

• CR: (landed in 1.12) Defined via OpenAPIv3 Schema grammar. more
• AA: You can customize any validating flow you want.

Conversion

• CR: (landed in 1.13) The CR conversioning (basically from storage version to requested version) could be done via conversioning webhook.
• AA: Develop any conversion you want.

SubResource

• CR: Currently only status and scale sub-resource supported.
• AA: You can customize any sub-resouce you want.

OpenAPI Schema

• CR: (landed in 1.13) The corresponding CRD’s OpenAPI schema will be automatically synced to root-apiserver’s openapi doc api.
• AA: OpenAPI doc has to be manually generated by code-generating tools.

Authentication

To make good use of AA, You need to be right about kubernetes And AA Have a certain understanding of the authentication mechanism , There are some concepts involved here

• Client certificate authentication
• token authentication
• Request header authentication

In the following instructions , All that appears APIServer All refer to Kubernetes Cluster components APIServer It can also be for root APIServer; be-all AA All refer to extension apiserver, It is self-developed AA.

Client certificate

The client certificate is CA Signed certificate , Specified by client CA certificate , Authenticate when the client connects , stay Kubernetes APIserver The same mechanism is used .

By default ,APIServer Specify parameters at startup --client-ca-file , At this time APIServer It creates a name extension-apiserver-authentication , Namespace is kube-system Under the configMap.

$ kubectl get cm -A
NAMESPACE     NAME                                 DATA   AGE
kube-system   extension-apiserver-authentication   6      21h

kubectl get cm extension-apiserver-authentication -n kube-system -o yaml

This can be seen from the above command configMap Will be populated to the client （AA Pod example ） in , Use this CA Certificates are used to authenticate clients CA. So the client will read this configMap, And APIServer Authentication .

I0622 14:24:00.509486       1 secure_serving.go:178] Serving securely on [::]:443
I0622 14:24:00.509556       1 configmap_cafile_content.go:202] Starting client-ca::kube-system::extension-apiserver-authentication::requestheader-client-ca-file

token authentication

Token Certification means passing HTTP Header Pass in Authorization: Bearer $TOKEN To authenticate the client , This is also Kubernetes Common methods of authentication in a cluster .

under these circumstances , Allow for APIServer Certification can also be applied to AA authentication . If not AA Authenticate the same cluster , or AA Running outside the cluster , The parameters can be --authentication-kubeconfig To specify the different... To use Kubeconfig authentication .

The following example is AA Start parameter of

./bin/apiserver -h|grep authentication-kubeconfig
      --authentication-kubeconfig string                        kubeconfig file pointing at the 'core' kubernetes server with enough righ
ts to create tokenreviews.authentication.k8s.io. This is optional. If empty, all token requests are considered to be anonymous and no cli
ent CA is looked up in the cluster.

Request header authentication

RequestHeader Certification means ,APIServer from AA Authentication by proxy connection .

By default ,AA from extension-apiserver-authentication Mentioned in ConfigMap in extract requestheader client CA Certificate and CN. If the Lord Kubernetes APIServer Configured options --requestheader-client-ca-file , Then it will fill in this content .

Skip client authentication --authentication-skip-lookup

to grant authorization

By default ,AA The server will automatically inject into Kubernetes Running on the cluster pod Connection information and credentials for , To connect to the master Kubernetes API The server .

E0622 11:20:12.375512       1 errors.go:77] Post "https://192.168.0.1:443/apis/authorization.k8s.io/v1/subjectaccessreviews": write tcp 192.168.0.36:39324->192.168.0.1:443: write: connection reset by peer

If AA Deploy outside the cluster , You can specify --authorization-kubeconfig adopt kubeconfig authentication , This is similar to information in binary deployment .

By default ,Kubernetes The cluster will enable RBAC, That means AA Create multiple clusterrolebinding.

The following log is AA In the case of resource access without permission in the cluster

E0622 09:01:26.750320       1 reflector.go:178] pkg/mod/k8s.io/[email protected]/tools/cache/reflector.go:125: Failed to list *v1.MutatingWebhookConfiguration: mutatingwebhookconfigurations.admissionregistration.k8s.io is forbidden: User "system:serviceaccount:default:default" cannot list resource "mutatingwebhookconfigurations" in API group "admissionregistration.k8s.io" at the cluster scope
E0622 09:01:29.357897       1 reflector.go:178] pkg/mod/k8s.io/[email protected]/tools/cache/reflector.go:125: Failed to list *v1.Namespace: namespaces is forbidden: User "system:serviceaccount:default:default" cannot list resource "namespaces" in API group "" at the cluster scope
E0622 09:01:39.998496       1 reflector.go:178] pkg/mod/k8s.io/[email protected]/tools/cache/reflector.go:125: Failed to list *v1.ValidatingWebhookConfiguration: validatingwebhookconfigurations.admissionregistration.k8s.io is forbidden: User "system:serviceaccount:default:default" cannot list resource "validatingwebhookconfigurations" in API group "admissionregistration.k8s.io" at the cluster scope

Need to manually namespace kube-system Created in rolebindding To role extension-apiserver-authentication-reader . So that you can access configMap 了 .

apiserver-builder

apiserver-builder A project is to create AA Tools for , You can refer to installing.md To install

Initialize project

Initialization command

• <your-domain> This is yours API Group of resources , Reference resources k8s.io/api
  • If the group name is a domain name, set it as the primary domain name , For example, the built-in Group
    • /apis/authentication.k8s.io
    • /apis/batch
• Generated go mod The package name is the name of your directory
  • for example , stay firewalld Under the table of contents ,go.mod For the name of the firewalld

apiserver-boot init repo --domain <your-domain>

for example

apiserver-boot init repo --domain fedoraproject.org

notes ： here –domain Set as the primary domain name , Later generated group Will follow the format +

apiserver-boot must be run from the directory containing the go package to bootstrap. This must
 be under $GOPATH/src/<package>.

Must be in $GOPATH/src Create your project under , I'm here for GOPATH=go/src , In this case, the project must be created in the directory go/src/src/{project} Create

Create a GVK

apiserver-boot create group version resource \
	--group firewalld \
	--version v1 \
	--kind PortRule

After the creation is completed, a api-like The type of , We just need to fill in what we need

type PortRule struct {
    
	metav1.TypeMeta   `json:",inline"`
	metav1.ObjectMeta `json:"metadata,omitempty"`

	Spec   PortRuleSpec   `json:"spec,omitempty"`
	Status PortRuleStatus `json:"status,omitempty"`
}

// PortRuleSpec defines the desired state of PortRule
type PortRuleSpec struct {
     //  The contents here are all empty , You can add it yourself 
	Name        string `json:"name"`
	Host        string `json:"host"`
	Port        int    `json:"port"`
	IsPremanent bool   `json:"isPremanent,omitempty"`
}

// PortRuleStatus defines the observed state of PortRule
type PortRuleStatus struct {
    
}

The generated code

apiserver-boot There are no special commands for generating code , You can execute any generation command , Here you use the generate binary executable command , This process is quite long .

apiserver-boot build executables

If compilation errors occur, you can use --generate=false Skip build , This can save a lot of time .

Operation mode

There are no more than three modes of operation , Run locally , Running in a cluster , Running outside the cluster

running_locally

Local operation requires a etcd service , Don't need to configure ca certificate , Use here docker function

docker run -d --name Etcd-server \
    --publish 2379:2379 \
    --publish 2380:2380 \
    --env ALLOW_NONE_AUTHENTICATION=yes \
    --env ETCD_ADVERTISE_CLIENT_URLS=http://etcd-server:2379 \
    bitnami/etcd:latest

And then execute the command , After successful execution, the corresponding access address will pop up

apiserver-boot build executables
apiserver-boot run local

running_in_cluster

Build a mirror image

You need to build the container image first ,apiserver-boot build container --image <image> This will generate code , structure apiserver and controller Binary , Then build the container image . After the construction is completed, the corresponding image needs to be push To the warehouse （ Optional ）

apiserver-boot build config \
	--name <servicename> \
	--namespace <namespace to run in> \
	--image <image to run>

notes , This operation needs to be supported in Linux Build in the kernel environment ,wsl There is no kernel function, so an error will be reported , Need to be replaced with wsl2, The tools are downloaded , if necessary wsl1+Docker Desktop structure , I need to modify myself

Building configuration

apiserver-boot build config \
	--name <servicename> \
	--namespace <namespace to run in> \
	--image <image to run>

The following steps are performed to build the configuration ：

• stay <project/config/certificates Create one in the directory CA certificate
• In the catalog <project/config/*.yaml Lower generation kubernetes List of resources required .

notes ：

In fact, this list is not perfect for any environment , You need to manually modify the configuration

Running Pod Contained in the apiserver And controller, If you use kubebuilder Created controller You can modify the resource list by yourself

modify apiserver Configuration of

The following parameters are about AA Certified parameters

--proxy-client-cert-file=/etc/kubernetes/pki/firewalld.crt \
--proxy-client-key-file=/etc/kubernetes/pki/firewalld.key \
--requestheader-allowed-names=kube-apiserver-kubelet-client,firewalld.default.svc,firewalld-certificate-authority \
--requestheader-client-ca-file=/etc/kubernetes/pki/front-proxy-ca.crt \
--requestheader-extra-headers-prefix=X-Remote-Extra- \

• --requestheader-username-headers： The header used to store the user name
• --requestheader-group-headers： Title for storage group
• --requestheader-extra-headers-prefix： Prefix appended to all additional headers
• --proxy-client-key-file ： Private key file
• --proxy-client-cert-file： Client certificate file
• --requestheader-client-ca-file： Signing the client certificate file CA Certificate
• --requestheader-allowed-names： Sign... In the client certificate CN)

It is known from the above information that , actually apiserver-boot The generated ca It will not be on , need kubernetes Their own ca To sign , Here are two simple commands , Use kubernetes Cluster certificates to issue certificates . here kubernetes Cluster certificate usage kubernetes-generator The production of . Here, according to this ca Generate again for AA Certified Certificate .

openssl req -new \
    -key firewalld.key \
    -subj "/CN=firewalld.default.svc" \
    -config <(cat /etc/pki/tls/openssl.cnf <(printf "[aa]\nsubjectAltName=DNS:firewalld, DNS:firewalld.default.svc, DNS:firewalld-certificate-authority, DNS:kubernetes.default.svc")) \
    -out firewalld.csr

openssl ca \
	-in firewalld.csr \
	-cert front-proxy-ca.crt \
	-keyfile front-proxy-ca.key \
	-out firewalld.crt \
	-days 3650 \
	-extensions aa \
	-extfile <(cat /etc/pki/tls/openssl.cnf  <(printf "[aa]\nsubjectAltName=DNS:firewalld, DNS:firewalld.default.svc, DNS:firewalld-certificate-authority, DNS:kubernetes.default.svc"))

Regenerate the required after completion yaml A list of resources is sufficient , Test the extended through the resource list API

apiVersion: firewalld.fedoraproject.org/v1
kind: PortRule
metadata:
  name: portrule-example
spec:
  name: "nginx"
  host: "10.0.0.3"
  port: 80

$ kubectl apply -f http.yaml 
portrule.firewalld.fedoraproject.org/portrule-example created
$ kubectl get portrule
NAME               CREATED AT
portrule-example   2022-06-22T15:12:59Z

For more detailed instructions, it is recommended to read Reference, Are official documents providing detailed instructions

Reference

aggregation layer

apiserver-builder doc