Sorry, you need to enable JavaScript to visit this website.

Home / Intel in Kubernetes* / Blogs / Itohan / 2020 / Using a Kubernetes* custom controller to manage two custom resources (Designing the Akraino* ICN BPA controller)

Using a Kubernetes* custom controller to manage two custom resources (Designing the Akraino* ICN BPA controller)

Author:
Itohan Ukponmwan
Last update:
Feb 05, 2020

Introduction

This article gives a brief overview of Kubernetes* (K8S*) custom resources and controllers and then describes in detail how to create a custom controller to manage two custom resources. Furthermore, this article also briefly describes how the Binary Provisioning Agent (BPA) controller required for the Akraino* Integrated Cloud Native (ICN) blueprint was built. It also discusses future plans and how the custom resources and controller discussed here can be extended to leverage certain IA hardware features.

Tutorials on creating custom controllers to manage a single custom resource abound. However, little to none are available on how to write custom controllers to manage two or more custom resources, this article aims to fill that void.

To understand this article, certain concepts must be described briefly;

  • Resource

    A resource is an endpoint in the Kubernetes API that stores a collection of API objects of a certain kind. For example, the built-in pods resource contains a collection of pod objects [1]. Other examples of built-in resources include deployments, jobs, configmaps, secrets, and others.

  • Custom Resource

    A custom resource (CR) is an extension of the Kubernetes API that is not necessarily available in a default Kubernetes installation. It represents a customization of a particular Kubernetes installation [1].

  • Custom Resource Definition

    A custom resource definition (CRD) is used to create the blueprint for a custom resource. In other words, it is used to define a new custom resource in a Kubernetes cluster. After a CRD is created, custom resource instances of that CRD can then be created.

  • Controller

    A controller tracks at least one Kubernetes resource type. These objects have a Spec field that represents the desired state. The controller(s) for that resource are responsible for making the current state come closer to that desired state [2].

  • Custom Controller

    A custom controller is used to manage a custom resource. It has the same function as the controller described in number four above, however it is not built-in. When a new custom resource is created, a custom controller should also be created to manage the custom resource and reconcile the current state of the CR with the desired state specified in the SPEC field of the CR API. Without a custom controller, custom resources just act as data storage.

  • Operator SDK

    Operator SDK is a framework that makes it easier to write operators (controllers). It generates boilerplate code and creates the folder structure.

BPA Controller Overview

The Binary Provisioning Agent (BPA) is a custom controller that is part of the Akraino Integrated Cloud Native (ICN) project, see wiki.akraino.org/pages/viewpage.action?pageId=11995877. The controller manages the provisioning custom resource and the software custom resource. It ultimately installs a particular Kubernetes deployment (KUD) in the nodes specified in the provisioning CR and then installs software specified in the software CR in the cluster specified.

When a resource of either kind is created, the Reconcile method of the BPA controller code is triggered and it performs functions based on the CR kind.

  1. Provisioning CR: If the created resource is a provisioning CR, the controller gets the cluster name, cluster type, list of MAC addresses for Master and Worker nodes respectively as well as the list of KUD plugins if any. Then it gets a list of bare metal hosts currently installed and confirms a host exists for the MAC addresses in the list. On confirming the host exists, the BPA controller uses the MAC address of each host to get its IP address from the DHCP lease file and creates a host.ini file. (KUD uses kubespray which uses the host.ini file to install Kubernetes.) After creating the host.ini file, the BPA controller starts a job to install KUD, it passes the host.ini file and the KUD plugins list (if any) to the job and spawns a thread that continuously checks the job status. If the job completed successfully, the BPA controller creates a Kubernetes configmap with the cluster name as a label and keeps a mapping of node name to IP address. It also adds a prefix to the node name to specify if the node is a worker or a master.

  2. Software CR: If the created resource is a software CR, the BPA controller gets the cluster name, a list of software to be installed in master nodes, and a list of software to be installed in worker nodes. Using the cluster name, it gets the configmap for that cluster (it throws an error if it does not find one) which contains the IP address of all hosts in the cluster as well as their roles. Finally, the BPA controller uses SSH to install software in each node depending on the roles.

Steps for Creating the Custom Resources and Custom Controller

The essential steps are described below. The full source code for the BPA controller can be found at github.com/akraino-edge-stack/icn/blob/master/cmd/bpa-operator/pkg/controller/provisioning/provisioning_controller.go and the entire BPA operator and its CRDs, APIs, etc. can be found at github.com/akraino-edge-stack/icn/tree/master/cmd/bpa-operator.

Prerequisites:

  • Golang and operator-sdk should be installed.
  • Check out the operator SDK user guide for a basic introduction to operator SDK.

1. Create a new bpa-operator project using the operator-sdk CLI:


   # operator-sdk new bpa-operator --repo=github.com/bpa-operator
   # cd bpa-operator
  

The above command creates the basic folder structure and also creates some files including the main.go file under the manager, which is the entry point for the operator. See the operator SDK user guide for more details.

Folder structure created:


 # tree ../bpa-operator
 ../bpa-operator/
 ├── build
 │ ├── bin
 │ │ ├── entrypoint
 │ │ └── user_setup
 │ └── Dockerfile
 ├── cmd
 │ └── manager
 │ └── main.go
 ├── deploy
 │ ├── operator.yaml
 │ ├── role_binding.yaml
 │ ├── role.yaml
 │ └── service_account.yaml
 ├── go.mod
 ├── go.sum
 ├── pkg
 │ ├── apis
 │ │ └── apis.go
 │ └── controller
 │ └── controller.go
 ├── tools.go
 └── version
 └── version.go
 

2. Create the custom resources.

  • Create the provisioning custom resource definition:

 # operator-sdk add api --api-version=bpa.akraino.org/v1alpha1 --kind=Provisioning

This puts the provisioning CR API in pkg/apis/bpa/v1alpha1/…

  • Create the software custom resource definition:

 # operator-sdk add api --api-version=bpa.akraino.org/v1alpha1 --kind=Software

This puts the software CR API in pkg/apis/bpa/v1alpha1/…

3. Define the spec and status for both custom resources.

  • Provisioning CR

Currently we do not keep status for the provisioning CR, so we modified the Spec and leave the status struct empty. The API file to modify is pkg/apis/bpa/v1alpha1/provisioning_types.go. When defining the specs, it is essential to think about the information the operator (controller) would need to carry out its functions. In this case, the operator needs the MAC addresses of the master and worker nodes to install KUD. It also needs a list of KUD plugins (if any) to be installed. In addition, in the future we want to be able to specify certain hardware requirements for the node such as CPU, memory, SR-IOV, etc. With that in mind, we modified the API file to include the requirements:


 type ProvisioningSpec struct {
 Masters []map[string]Master `json:"masters,omitempty"`
 Workers []map[string]Worker `json:"workers,omitempty"`
 KUDPlugins []string `json:"KUDPlugins,omitempty"`
 }
 type Provisioning struct {
 metav1.TypeMeta `json:",inline"`
 metav1.ObjectMeta `json:"metadata,omitempty"`
 Spec ProvisioningSpec `json:"spec,omitempty"`
 Status ProvisioningStatus `json:"status,omitempty"`
 }
 type ProvisioningList struct {
 metav1.TypeMeta `json:",inline"`
 metav1.ListMeta `json:"metadata,omitempty"`
 Items []Provisioning `json:"items"`
 }
 // master struct contains resource requirements for a master node
 type Master struct {
 MACaddress string `json:"mac-address,omitempty"`
 CPU int32 `json:"cpu,omitempty"`
 Memory string `json:"memory,omitempty"`
 }
 // worker struct contains resource requirements for a worker node
 type Worker struct {
 MACaddress string `json:"mac-address,omitempty"`
 CPU int32 `json:"cpu,omitempty"`
 Memory string `json:"memory,omitempty"`
 SRIOV bool `json:"sriov,omitempty"`
 QAT bool `json:"qat,omitempty"`
 }
 

The full API file can be found in the icn repo. The ProvisioningSpec struct contains a list of Masters, Workers, and KUDPlugins.

  • Software CR

We also do not keep status for this CR, so we modified the Spec struct as required. We used a list of type interface for the software list because the list can contain either just the software name as a string or it can be a map of a map containing the software name and the desired version.


 type SoftwareSpec struct {
 MasterSoftware []interface{} `json:"masterSoftware,omitempty"`
 WorkerSoftware []interface{} `json:"workerSoftware,omitempty"`
 }
 

4. The .yaml CRD and CR files are created in deploy/crds….

The default CRDs (bpa_v1alpha1_provisioning_crd.yaml and bpa_v1alpha1_software_crd.yaml) are suitable for both the software CR and the provisioning CR. We added the shortNames parameter and value to both files so shorter names can be used with kubectl when using the resource. On the other hand, the CR files must be modified significantly to correspond to the API. You must create a new CR file whenever you want to create an instance of the CRDs. A sample of each CR is explained below.

  • Provisioning

 apiVersion: bpa.akraino.org/v1alpha1
 kind: Provisioning
 metadata:
 name: sample-kud-plugins
 labels:
 cluster: cluster-efg
 owner: c2
spec:
 masters:
 - master-1:
 mac-address: 00:e1:ba:ce:df:bd
 workers:
 - worker-1:
 mac-address: 00:c4:13:04:62:b5
 KUDPlugins:
 - onap4k8s

Each item directly under spec (masters, workers, KUDPlugins) corresponds to a parameter under ProvisioningSpec in the API.

  • Software

apiVersion: bpa.akraino.org/v1alpha1
kind: Software
metadata:
 labels:
 cluster: cluster-xyz
 owner: c1
 name: sample-software
spec:
 masterSoftware:
 - expect
 - htop
 - jq:
 version: 1.5+dfsg-1ubuntu0.1
 - maven
 workerSoftware:
 - curl
 - python
 - tmux
 - jq

5. Add a controller to the bpa-operator project. This controller watches and reconciles the provisioning custom resource (and later the software custom resource as well).


 operator-sdk add controller --api-version=bpa.akraino.org/v1alpha1 --kind=Provisioning

This command creates boilerplate code for the controller in pkg/controller/provisioning/provisioning_controller.go so we can modify this file.

This step is where most of the customization magic happens. The operator SDK creates the basic functions and we modify each as required:


 provisioningInstance := &bpav1alpha1.Provisioning{}
 softwareInstance := &bpav1alpha1.Software{}
 err := r.client.Get(context.TODO(), request.NamespacedName, provisioningInstance)
 provisioningCreated := true
 if err != nil {
 //Check if its a Software Instance
 err = r.client.Get(context.TODO(), request.NamespacedName, softwareInstance)
 if err != nil {
 if errors.IsNotFound(err) {
 // Request object not found, could have been deleted after reconcile request.
 // Owned objects are automatically garbage collected. For additional cleanup logic use finalizers.
 // Return and don't requeue
 return reconcile.Result{}, nil
 }
 // Error reading the object - requeue the request.
 return reconcile.Result{}, err
 }
 //No error occured and so a Software CR was created not a Provisoning CR
 provisioningCreated = false
 }

The code snippet shows the logic that determines if the custom resource that triggered the Reconcile loop is a provisioning CR or a software CR. When the Reconcile loop is triggered, the BPA operator creates variables for a provisioning instance and a software instance. Within the Reconcile loop, the BPA operator first checks if the Reconcile loop was triggered by a provisioningInstance by performing a Get request using the client and trying to put the output of the Get request in the provisioning instance variable. If no error occurs, the BPA operator confirms that a provisioning instance was created and proceeds to the logic for the provisioning instance. If an error occurs, the BPA operator then checks if the Reconcile loop was triggered by a software instance also by perofrming a Get request using the client and trying to put the output in the Software Instance variable. If no error occurs (error == nil) at this call, the BPA operator confirms that a software instance was created and proceeds to the logic for the software instance.

On the other hand, if an error occurs and the error says the instance was not found, the BPA operator assumes that a delete request triggered the Reconcile loop and so the reconcile was successful, so it does not requeue the request for Reconcile. Finally, if none of the cases above were met, then an actual error occured and the BPA operator will requeue the request for Reconcile.

After the above steps, it proceeds as described in BPA Controller overview above depending on the CR Kind created. The full source code is available here.

  • The add function which contains calls to the watch methods. The watch method watches for changes to the resources specified and calls the Reconcile loop. For more details, see resources watched by the controller. Here we watch for changes to the provisioning CR, software CR (which are the primary resources), and Job resource and configmap resource (which are secondary resources created by the provisioning CR logic).
  • The Reconcile loop is called with the ReconcileProvisioning struct. This struct contains variables used in the Reconcile loop. We will see how these are used as we go further. By default, operator-sdk creates the struct with client and scheme variables. We added one more variable:
    • bmhClient dynamic interface which is used to list all bare metal hosts custom resources in the cluster.
    
     type ReconcileProvisioning struct {
     // This client, initialized using mgr.Client() above, is a split client
     // that reads objects from the cache and writes to the apiserver
     client client.Client
     scheme *runtime.Scheme
     bmhClient dynamic.Interface
     }
    

    Note: The code described here is slightly different from the code in the Akraino icn repo. I am currently working on bug fixes and refactoring, the updated code has not been merged in the ICN repo but is available in my git repo.

  • The newReconciler function creates an instance of the ReconcileProvisioning struct with all variables populated.
  • The Reconcile method is the meat of it all. This is where the Reconcile loop happens and other subsequent functions are just helper functions to be used in this method. Whenever the Reconcile method is called, the first thing is to fetch the custom resource instance that triggered the Reconcile loop and determine if it is a provisioning CR or a software CR. The code snippet is shown below:

6. An image is built from the provisioning_controller.go code and the controller is run as a pod. However, while testing, the operator can be run using an operator-sdk command:


 # operator-sdk up local --kubeconfig ~/.kube/config --verbose
 

7. When we are ready to run the operator, we build the images required in the following way:

  • Modify build/Dockerfile

 FROM registry.svc.ci.openshift.org/openshift/release:golang-1.12 AS builder
 WORKDIR /go/src/github.com/bpa-operator
 COPY . .
 RUN make build
 FROM registry.svc.ci.openshift.org/openshift/release:golang-1.12
 COPY --from=builder
 /go/src/github.com/bpa-operator/build/_output/bin/bpa-operator /
  • Download dependencies into vendor directory:
# go mod vendor
  • Build the images. We build the BPA operator image and the multicloud-k8s image required for the KUD installer job:
docker build --rm -t akraino.org/icn/bpa-operator:latest . -f build/Dockerfile
 git clone https://github.com/onap/multicloud-k8s.git
 cd multicloud-k8s && \
 docker build --network=host --rm \
 --build-arg http_proxy=${http_proxy} \
 --build-arg HTTP_PROXY=${HTTP_PROXY} \
 --build-arg https_proxy=${https_proxy} \
 --build-arg HTTPS_PROXY=${HTTPS_PROXY} \
 --build-arg no_proxy=${no_proxy} \
 --build-arg NO_PROXY=${NO_PROXY} \
 --build-arg KUD_ENABLE_TESTS=true \
 --build-arg KUD_PLUGIN_ENABLED=true \
 -t github.com/onap/multicloud-k8s:latest . -f kud/build/Dockerfile
  • Operator-sdk creates deployment files (service_account.yaml, role.yaml, role_binding.yaml, operator.yaml) and modifies each of the files to meet our requirements. For example, we change role and role_binding to cluster role and cluster role binding. We also modify the other files, see repo for specifics. After that, we create the CRDs, secret (required by KUD installer job) and the deployment using the following commands:
#kubectl apply -f deploy/service_account.yaml
#kubectl apply -f deploy/role.yaml
#kubectl apply -f deploy/role_binding.yaml
#kubectl apply -f deploy/crds/provisioning-crd/bpa_v1alpha1_provisioning_crd.yaml
#kubectl apply -f deploy/crds/software-crd/bpa_v1alpha1_software_crd.yaml
#kubectl apply -f deploy/operator.yaml
#kubectl create secret generic ssh-key-secret --from-file=id_rsa=/root/.ssh/id_rsa --from-file=id_rsa.pub=/root/.ssh/id_rsa.pu

Note: The repo has a makefile that handles image build and the above steps. This can be done with one command:


 # make docker && make deploy

 

  • The operator is now running and ready to reconcile instances of the provisioning CR and the software CR:

 #kubectl get deployment
 NAME READY UP-TO-DATE AVAILABLE AGE
 bpa-operator 1/1 1 1 84s
 #kubectl get pods
 NAME READY STATUS RESTARTS AGE
 bpa-operator-5b56675ccf-4fvct 1/1 Running 0 112s
  • Example provisioning CR instance and sample logs Sample provisioning CR yaml:

 apiVersion: bpa.akraino.org/v1alpha1
 kind: Provisioning
 metadata:
 name: e2e-test-provisioning
 labels:
 cluster: cluster-test
 owner: c1
 spec:
 masters:
 - master-test:
 mac-address: 08:00:27:00:ab:c0

Create the above instance:


#kubectl create -f sample-provisioning.yam
#kubectl get pods
NAME READY STATUS RESTARTS AGE
bpa-operator-5b56675ccf-4fvct 1/1 Running 0 70m
kud-cluster-test-w4xhw 1/1 Running 0 3m
#kubectl logs bpa-operator-5b56675ccf-4fvct
BareMetalHost CR bpa-test-bmh has NIC with MAC Address 08:00:27:00:ab:c0
192.168.50.63 : 08:00:27:00:ab:c0
Checking job status for cluster cluster-test

Summary

Kubernetes custom controllers and custom resources provide a powerful way to extend the functionality of the Kubernetes API while the operator SDKframework makes creating these controllers and resources much easier as it creates boilerplate code. In this article, we used the custom controller to manage two custom resources, however it can be extended to manage three or more depending on the use case.  You can download the latest code from my github repo or the upstream code from the Akraino ICN repo (does not have latest bpa operator code as of writing this) and test as well as use it as a starting point to writing controllers for more custom resources.

We welcome ideas for further enhancements.

Comments/Questions? Join the Akraino-icn slack channel.

Future work

Currently, when a provisioning instance is deleted, KUD is not uninstalled from the cluster. We plan to add a finalizer so when a delete request is made to delete a provisioning instance, the instance will not be deleted until KUD has been uninstalled from the cluster.

References

  1. https://kubernetes.io/docs/concepts/extend-kubernetes/api-extension/custom-resources/
  2. https://kubernetes.io/docs/concepts/architecture/controller/
  3. https://github.com/operator-framework/operator-sdk