Sorry, you need to enable JavaScript to visit this website.
Home / Intel in Kubernetes* / Blogs / Chengli3 / 2020 / Implement Label-based Permission Control for Kubernetes* Using Webhook

Implement Label-based Permission Control for Kubernetes* Using Webhook

Author:
Cheng Li
Last update:
Jun 23, 2020

Introduction

Kubernetes supports permission control on a namespace level. However, sometimes this doesn’t satisfy our requirement. For example, in the Akraino ICN project, we have some advanced permission requirements: Multiple users can create, update, or delete one kind of resource in the same namespace, but they can’t update or delete objects created by others. To address this requirement, we designed and implemented a label-based permissions control mechanism in Kubernetes by using a webhook. Normally, webhooks are used to validate the resources or set default values for resources. In this article, we use webhooks to control permissions.

Design of the permission system

In Kubernetes, user or service accounts can be bound to one or more roles. Each role defines its permission rules. For example, the following definition claims that the sdewan-test role can create or update custom resource instances (CRs) of type Mwan3Rule in the default namespace and can get Mwan3Policy CRs.


apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
    Annotations:
    name: sdewan-test
    namespace: default
rules:
- apiGroups:
    - ""
    resources:
    - mwan3rules
    verbs:
    - create
    - update
- apiGroups:
    - ""
    resources:
    - mwan3policies
    verbs:
    - get

    We extend the role with annotations sdewan-bucket-type-permission, which is in json format. In the annotation, we can define label-based permissions. For example, the role shown below extends the sdewan-test role permission: sdewan-test can only create/update Mwan3Rule (the json key specifies the customized resource type) CRs with the label sdewan-bucket-type=app-intent or sdewan-bucket-type=k8s-service. Also it can only get Mwan3Policy CR with the label sdewan-bucket-type=app-intent. Actually, we support wildcard matches as well. For example, we can use mwan3* to involve both mwan3policies and mwan3rules.

    
    apiVersion: rbac.authorization.k8s.io/v1
    kind: Role
    metadata:
      annotations:
        sdewan-bucket-type-permission: |-
          { "mwan3rules": ["app-intent", "k8s-service"],
          "mwan3policies": ["app-intent"] }
      name: sdewan-test
      namespace: default
    rules:
    - apiGroups:
      - ""
      resources:
      - mwan3rules
      verbs:
      - create
      - update
    - apiGroups:
      - ""
      resources:
      - mwan3policies
      verbs:
      - get
    

      Kubernetes webhook is responsible for parsing the role annotations. Let me describe what's an admission webhook in simple words. We can think webhook as a web service, normally it runs as a pod in the Kubernetes cluster. When the Kubernetes api receives a request, kube-api can call the webhook API before saving the object into etcd. If the webhook returns `allowed=true`, kube-api continues to persist the object into etcd. Otherwise, kube-api rejects the request.

      The Webhook request body has a field named userInfo that indicates who is making the Kubernetes api request. From the userInfo, webhook can get the role information, then get the role annotation. By comparing the label-level permission described in the annotation with the CR label, the webhook can decide whether to allow the request.

      Implementation of the permission control system

      We have implemented the label-based permission system through a kubebuilder framework in the Akraino ICN project. The assumption here is that some Custom Resource Definitions (CRDs) (such as Mwan3Policy and Mwan3Rule) and the corresponding controllers have already been implemented.

      Kubebuilder can generate the basic CRD, controller code, and webhook code. To create a new webhook, we ran the following kubebuilder command:

      
      kubebuilder create webhook --group batch --version v1 --kind CronJob --programmatic-validation
      

         

        This command leverages the controller-runtime builder to create a validation webhook. The command also creates the webhook server, which accepts https requests from the kube-api server. It even generates the code that parses the http request body and converts it into a CRD structure instance. The developers just need to write code to validate the CRD instance content. This is good for most webhook cases. But it doesn’t satisfy our requirements, because we need the UserInfo which is in the https request body instead of the CRD instance.

        We reused the webhook server generated by kubebuilder, but implemented the handler by ourselves. The diagram above shows the handler process flow. The handler accepts “admission.Request” as input, which contains UserInfo. In the handler, we get the user role information and the CR label.

        • Get the user role information.

          The userinfo is in the request body. To get roles information from the userinfo, the handler has to ask kube-api for the role information of the user. The handler sends requests to kube-api with the serviceaccount token as authentication. First, the handler sends one request to get rolebinding information of the user. Then it sends the second request to get the role information from the rolebinding information. By default, the rolebinding resources don’t have an indexer on the ".subjects” field. It means that we can’t filter rolebinding in a kube-api request, unless we add the following indexer for rolebinding:
                       err = mgr.GetFieldIndexer().IndexField(context.Background(), &rbacv1.RoleBinding{}, ".subjects", func(rawObj runtime.Object) []string {
                        var fieldValues []string
                        rolebinding := rawObj.(*rbacv1.RoleBinding)
                        for _, subject := range rolebinding.Subjects {
                                if subject.Kind == "ServiceAccount" {
                                        fieldValues = append(fieldValues, fmt.Sprintf("system:serviceaccount:%s:%s", subject.Namespace, subject.Name))
                                } else {
                                        fieldValues = append(fieldValues, subject.Name)
                                }
                        }
                        return fieldValues
                })
        
          • Get the CR information

            The request from users could be creating, updating, or deleting a resource. For create or update requests, the handler extracts the CR information from the request body. For delete requests, the handler can only get the CR name but not the whole instance. So it needs to call kube-api to get the CR information
          • Considering Role and ClusterRole

            Besides Role, user permission could be also defined in ClusterRole. The difference is that Role always defines the permission in a single namespace but ClusterRole can define the permission in the whole cluster. So we also need to check the annotations in the ClusterRole. When parsing Role information, we only care about the roles that have the same namesapce with the CR.

          Test and experiment

          Now I would like to introduce how to test and experiment with the label-based permission. In ICN, this feature is developed through kubebuilder, together with some CRDs/controllers.

          Webhook supports https but not http. So we need a certification for the webhook server. We use the cert-manager plugin to populate certificates for the webhook server. At the same time, we also inject the certificates into the webhook configuration so that the kube-api can use the certificates when sending webhook requests. For development or local test cases, where webhook runs from local instead of pod, we need to configure certificates by hand.

          Once the webhook is up, we can create roles and rolebindings for tests. Let’s create the following role and bind it to a user.

          
          apiVersion: rbac.authorization.k8s.io/v1
          kind: Role
          metadata:
            namespace: default
            name: create-intent
            annotations:
              sdewan-bucket-type-permission: |-
                { "mwan3policies": ["app-intent"] }
          rules:
          - apiGroups: ["batch.sdewan.akraino.org"]
            resources: ["mwan3policies"]
            verbs: ["get", "watch", "list", "delete", "create"]
          

             

            The user is allowed to create mwan3policies of label “sdewan-bucket-type: app-intent” but not other labels. A concrete example here would be the user can create the left CR but not the right one. If we are trying to create the right CR, we will get the error message `Error from server (Your roles don't have the permission): error when creating "config/samples/batch_v1alpha1_mwan3policy.yaml"`

             

            apiVersion: batch.sdewan.akraino.org/v1alpha1
            kind: Mwan3Policy
            metadata:
              name: balance1
              namespace: default
              labels:
                sdewanPurpose: cnf1
                sdewan-bucket-type: app-intent
            spec:
              members:
                - network: ovn-net1
                  weight: 4
                  metric: 2
                - network: ovn-net2
                  weight: 3
                  metric: 3
                        
            apiVersion: batch.sdewan.akraino.org/v1alpha1
            kind: Mwan3Policy
            metadata:
              name: balance1
              namespace: default
              labels:
                sdewanPurpose: cnf1
                sdewan-bucket-type: infra-intent
            spec:
              members:
                - network: ovn-net1
                  weight: 4
                  metric: 2
                - network: ovn-net2
                  weight: 3
                  metric: 3
                        

            Conclusion

            Webhook is a very good feature of kubernetes, it grants kubernetes more flexibility. Developers can use it to implement many useful features. Controller-runtime project provides the builder tool with which we can create two types of webhook easily: Validating Webhook and Mutating Webhook. Validating Webhook is used to validate the resources for kube-api and Mutating Webhook can update values of resource fields, e.g. setting default values. Sometimes these two types of webhook don’t fit our requirements. For example, in this article we need to get the userinfo which is not in the resource body. If that is the case, we need to develop most of the handler code by ourselves instead of using the builder tool.

            Links for reference