Sorry, you need to enable JavaScript to visit this website.

Integrating Intel® QAT into Kubernetes* VMs, Part 1

BY Le Yao ON Jul 30, 2020

Introduction

Kubernetes* is an open-source container-orchestration system for automating application deployment, scaling, and management. It was originally designed by Google*, and is now maintained by the Cloud Native Computing Foundation*.

Akraino* is an open-source software stack that improves the state of edge cloud infrastructure for carrier, provider, and IoT networks based on the Kubernetes cluster. The Integrated Cloud Native NFV/App stack family (ICN) is one of its blueprints. The ICN Blueprint (BP) family intends to address deployment of workloads in a large number of edges and also in public clouds using Kubernetes as the resource orchestrator in each site and ONAP-K8S as the service level orchestrator (across sites). ICN also intends to integrate infrastructure orchestration which is needed to bring up a site using bare-metal servers.In ICN, some services required by user cases can not easily be contained. So we need to launch a virtual machine (VM) in Kubernetes to take advantage of applications that can’t run in containers.

This article introduces the device plugin of Kubernetes, the background of Intel® QuickAssist Technology (Intel® QAT), how to launch VMs into a Kubernetes cluster, how to integrate Intel QAT VFs into these VMs, and take advantage of them.

Device plugin

Overview

Kubernetes provides vendors with a mechanism called a device plugin. Device plugins are a kind of simple gRPC server that may run in a container deployed through the pod mechanism or run on the host. A related concept for device plugins is Kubernetes extended-resources. By sending a REST request via standard HTTP verbs (PATCH here) to the Kubernetes API server, a custom resource type is added to the node. This is used for the quota statistics of the resource and the corresponding QoS configuration.

To send a patch node request conveniently, start a proxy so you can easily send requests to the Kubernetes API server. First, execute a kube proxy command to start it temporarily, then add six intel.com/devices resource to a node (~1 in the commands will automatically transform into /):

curl --header "Content-Type: application/json-patch+json" \
--request PATCH \
--data '[{"op": "add", "path": "/status/capacity/intel.com~1devices", "value": "6"}]' \
http://localhost:8001/api/v1/nodes/<your-node-name>/status

Then extend six intel.com/devices resources for the node, and you can see the following:

kubectl describe node xxx
...
Capacity:
  ephemeral-storage:   3650656984Ki
  cpu:                 72
  memory:              263895388Ki
  intel.com/devices:   6
 pods:                 110
...

We can use these resources in our pod by adding intel.com/devices: "1" to spec.containers.resources.requests/limits and the pod is scheduled with statistics.

To clean up the extended resources, execute the following command:

curl --header "Content-Type: application/json-patch+json" \
--request PATCH \
--data '[{"op": "remove", "path": "/status/capacity/intel.com~1devices"}]' \
http://localhost:8001/api/v1/nodes/<your-node-name>/status

Functionality

DevicePlugin is the service advertised by device plugins in a Kubernetes cluster:

service DevicePlugin {
      // returns a stream of []Device
      rpc ListAndWatch(Empty) returns (stream ListAndWatchResponse) {}
      rpc Allocate(AllocateRequest) returns (AllocateResponse) {}
}

Device plugins can finish the following three tasks for the vendors:

  • Advertise devices.
  • Monitor devices (currently, by performing health checks).
  • Hook into the runtime to execute device specific instructions and to make the device available in the container.

Figure 1. The workflow of the device plugin

Why we use device plugins

  • Very few devices are handled natively by Kubelet (cpu and memory).
  • Device plugins provide a sustainable solution for vendors to be able to advertise their resources to Kubelet and monitor them without writing custom Kubernetes code.
  • Device plugins are consistent and portable solutions to consume hardware devices across k8s clusters that use a particular device type (GPU, Intel QAT, FPGA, etc.) in pods.

How Device Plugin works

In Kubernetes, Kubelets offer a register gRPC server that allows the device plugin to register itself to Kubelet. When starting, the device plugin makes a (client) gRPC call to the Register function that Kubelet exposes. The device plugin sends a RegisterRequest to Kubelet to notify Kubelet of the following information:

  1. Its own unix socket name, which receives the requests from Kubelet through the gRPC apis.
  2. The api version of device plugin itself
  3. The resource name they want to advertise. The resource name must follow a specified format (vendor-domain/vendor-device). such as intel.com/qat

The Kubelet responds to the RegisterRequest with a RegisterResponse that contains any error Kubelet might have encountered (such as an API version that is not supported or a resource name already registered). Then, the device plugin starts its gRPC server if it did not receive an error.

After successful registration, Kubelet calls the ListAndWatch function from the device plugin. A ListAndWatch function is for the Kubelet to discover the devices and their properties as well as to notify of any status change (such as if devices become unhealthy). The list of devices is returned as an array of the description information of all the devices (such as ID or health status) of the resource. Kubelet records this resource and its corresponding number of devices to node.status.capacity/allocable and updates it to apiserver. This function always does a loop check. Once the device is abnormal or unplugged from the machine, it updates and returns the latest device list to Kubelet.

In this way, when creating a pod, fields such as intel.com/qat can be added to spec.containers.resource.limits/requests to inform Kubernetes to schedule the pod to nodes with more than one intel.com/qat resource allowance. When the pod is to run, Kubelet calls the device plugin allocation function. The device plugin may do some initialization operations, such as QAT configuration or QRNG initialization. If initialization is successful, this function returns the necessary information for the pod to utilize the assigned device, and the information is passed to the container runtime as a parameter used to run the container. The information returned by the device plugin is shown below:

message ContainerAllocateResponse {
    map<string, string> envs // Env vars added to container runtime
    repeated Mount mounts // Mount info
    repeated DeviceSpec devices // Device Info
    map<string, string> annotations
}
message Mount {
    string container_path // Path in container
    string host_path // Path in host
    bool read_only
}
message DeviceSpec {
    string container_path
    string host_path
    string permissions // The permission needed
}

User flow

To use the extended resource, we add intel.com/qat to spec.containers.resource.limits/requests. We expect the request to have limits == requests. Here are the typical steps users follow to use the extended resource:

  1. A user submits a pod spec requesting one Intel QAT through intel.com/qat.
  2. The scheduler filters the nodes that do not match the resource requests.
  3. The pod lands on the node and Kubelet decides which device should be assigned to the pod.
  4. Kubelet calls Allocate on the matching Device Plugins.
  5. The user deletes the pod or the pod terminates.

 

When receiving a pod that requests devices, Kubelet is in charge of:

  • Deciding which device to assign to the pod's containers
  • Calling the Allocate function with the list of devices

The Kubernetes scheduler is in charge of filtering the nodes that cannot satisfy the resource requests.

Intel QAT

Introduction

Intel QuickAssist Technology (Intel QAT) is developed by Intel and runs on Intel® architecture providing security and compression acceleration capabilities to improve performance and efficiency. It offloads workloads from the CPU to hardware. Server, networking, big data, and storage applications use Intel QAT to offload compute-intensive operations, such as:

  • Symmetric cryptography functions, including cipher operations and authentication operations.
  • Public key functions, including RSA, Diffie-Hellman, and elliptic curve cryptography.
  • Compression and decompression functions, including DEFLATE.

Intel QAT has improved the function of in many areas, such as Hadoop* acceleration, OpenSSL integration, SDN and NFV Solutions Boost and so on. On a more detailed level, we can get the benefits from the following aspects:

  • 4G LTE and 5G encryption algorithm offload for mobile gateways and infrastructure.
  • VPN traffic acceleration, with up to 50 Gbps crypto throughput and support for IPsec and SSL acceleration.
  • Compression/decompression, with up to 24 Gbps throughput.
  • I/O virtualization using PCI-SIG Single-Root I/O Virtualization (SR-IOV).

Hardware

Intel QAT has three forms in actual use scenarios:

  • Chipset: Intel® C6xx series chipset
  • PCIE: Intel® QuickAssist Adapter 89xx
  • SoC: Intel Atom® C3000 processor series (Denverton NS) / Rangeley

Features

To improve the user experience, Intel QAT has optimized the design in terms of function and usage:

  • Intel QAT provides security (encryption) HW acceleration and compression HW acceleration
  • Intel QAT uses a set of APIs to abstract the hardware. The same application can run on multiple generations of Intel QAT hardware
  • Customers can also make use of patches that Intel QAT provides to popular open source software, so they can minimize or eliminate their effort to learn the API

With the support above, Intel QAT makes it easier for developers to integrate the accelerators in their designs and thus decrease the development time. It can increase business flexibility by offering solutions that best fit the changing business requirements. It also frees up valuable cycles on processors and allows it to perform value-added functionality.

What's more, Intel QAT provides a uniform means of communication between accelerators, applications, and acceleration technologies. Due to this, the resources are managed more productively. Then It can boost application throughput by reducing the demand on the platform and maximizing the CPU utilization.

Deploy

To make use of the Intel QAT device in the Kubernetes cluster, we need to deploy the Intel QAT device plugin into our cluster. Because we want to assign Intel QAT virtual function (VF) to the VMs managed by Kubernetes, we need to obtain the device id of the specific device from the Intel QAT device plugin. Only the Data Plane Development Kit (DPDK) mode in the Intel QAT device plugin exposes the device id to Kubernetes.

First, we need to compile the Linux Kernel QAT driver in our host and bind vfio-pci drivers to the VFs. Before we apply the QAT device plugin yaml file, we need to create a Kubernetes configmap for the device plugin to detect the QAT VFs and advertise them to Kubernetes.  The following is a simple example of the configmap.

apiVersion: v1
kind: ConfigMap
metadata:
  name: intel-qat-plugin-config
data:
  DPDK_DRIVER: "vfio-pci"
  KERNEL_VF_DRIVERS: "dh895xccvf,c6xxvf,c3xxxvf,d15xxvf"
  MAX_NUM_DEVICES: "32"
  VERBOSITY: "0"

Then we can setup the device plugin into our cluster. It detects the Intel QAT VFs bound with the related drivers defined in the configmap. Then we can easily get the devices in the cluster and assign them to the target pod or VM . The following is an example of the Intel QAT device plugin DaemonSet.

apiVersion: apps/v1
kind: DaemonSet
metadata:
  name: intel-qat-plugin
  labels:
    app: intel-qat-plugin
spec:
  selector:
    matchLabels:
      app: intel-qat-plugin
  template:
    metadata:
      labels:
        app: intel-qat-plugin
    spec:
      containers:
      - name: intel-qat-plugin
        image: intel/intel-qat-plugin:devel
        securityContext:
          readOnlyRootFilesystem: true
        env:
        - name: DPDK_DRIVER
          valueFrom:
            configMapKeyRef:
              name: intel-qat-plugin-config
              key: DPDK_DRIVER
        - name: KERNEL_VF_DRIVERS
          valueFrom:
            configMapKeyRef:
              name: intel-qat-plugin-config
              key: KERNEL_VF_DRIVERS
        - name: MAX_NUM_DEVICES
          valueFrom:
            configMapKeyRef:
              name: intel-qat-plugin-config
              key: MAX_NUM_DEVICES
        - name: VERBOSITY
          valueFrom:
            configMapKeyRef:
              name: intel-qat-plugin-config
              key: VERBOSITY
 
        imagePullPolicy: IfNotPresent
        args: ["-dpdk-driver", "$(DPDK_DRIVER)", "-kernel-vf-drivers", "$(KERNEL_VF_DRIVERS)", "-max-num-devices", "$(MAX_NUM_DEVICES)", "-v", "$(VERBOSITY)"]
        volumeMounts:
        - name: devdir
                              mountPath: /dev/vfio
          readOnly: true
        - name: pcidir
          mountPath: /sys/bus/pci
        - name: kubeletsockets
          mountPath: /var/lib/kubelet/device-plugins
      volumes:
      - name: devdir
        hostPath:
          path: /dev/vfio
      - name: pcidir
        hostPath:
          path: /sys/bus/pci
      - name: kubeletsockets
        hostPath:
          path: /var/lib/kubelet/device-plugins

Launch VMs in Kubernetes

Establishing a VPN requires confidentiality, integrity, and authentication through cryptography. IPSec or Datagram Transport Layer Security (DTLS) protocol can use a VPN to accelerate the compute-intensive process involved.

In ICN, IPSec is used to connect different edge nodes and traffic hubs through WAN. Because of different scenarios and requirements, both cloud-native network function (CNF) and virtual network function (VNF) are needed in ICN. To maximize the CPU utilization, we need to offload the workload from CPU to Intel QAT.

We need virtual machines to set up VNFs. As we stated in the introduction, Kubernetes is an open-source container-orchestration system for automating application deployment, scaling, and management. It can’t hold the virtual machines and manage them like a plain pod. So the open source community proposes many different ways to integrate the VMs into the Kubernetes cluster. We focus on the following two solutions:

Virtlet is a Kubernetes container runtime interface (CRI) implementation for running VM-based pods on Kubernetes clusters. CRI is what enables Kubernetes to run non-Docker* flavors of containers, such as Rkt.

KubeVirt is a Kubernetes custom resource definitions (CRD) implementation. KubeVirt is a virtual machine management add-on for Kubernetes. The aim is to provide a common ground for virtualization solutions on top of Kubernetes.

  • Virtlet
  • Kubevirt

Summary

At the beginning, this article introduced what the Device Plugin is and why we need it in Kubernetes. The article described the workflow of the device plugin and explained how a PCI device VF is assigned to a container or a vm. Then we introduced Intel QAT and how to take advantage of it. Finally, we mentioned two kinds of solutions to launch VMs in Kubernetes.

In conclusion, we introduced the background of the work we have finished and in the next articles, we will show more details about Intel QAT enabling in Virtlet and Kubevirt.

References