Sorry, you need to enable JavaScript to visit this website.

Pedal to the Metal: Accelerator Configuration and Control for Open Source

BY JING LIN, Dave Jiang, Tony Luck ON May 14, 2020

Abstract

While accelerators are commonly found in data centers and in the cloud, a new class of acceleration devices is coming. These new devices offload work from the main CPU and have major differences from previous generations of accelerators: they can operate on data based on virtual addresses rather than directly on physical addresses, they are directly accessible by application code in user space, they can be transparently shared by multiple applications, containers, or VMM guests, and they are highly configurable for different needs in different scenarios. The Intel® Data Streaming Accelerator (Intel® DSA) is one such accelerator that performs high speed data-copy and data transformation in Intel® Xeon® processors. In this article, we introduce the tool accel-config, which allows you to configure the accelerator device from within a user application in an easy and scalable way.

Problem

On a cloud platform with a large number of accelerators shared among multiple containers and VMM guests, you need a straightforward mechanism for discovering the capabilities of these complex devices and configuring them. The kernel boot command line or module parameters do not provide enough flexibility for you to easily accomplish configuration. Through the kernel device driver, these devices enumerate capabilities via a hierarchy of files in the /sys file system. These same files exposed through sysfs can be written to configure the devices. The command line tool should provide an additional layer of user-friendliness for you to configure accelerator devices. All the features of the tool are also available in a library for direct use by applications that want to provide their own user interface to use accelerators.

Solution

The idxd kernel driver (which serves as a driver for Intel DSA) has been developed to utilize the Linux* kernel dmaengine subsystem to service kernel work requests. The kernel driver also provides a direct submission path to the DSA devices for low-latency work requests. The driver offers a character driver to export the user command submission portal via mmap.

We also implemented the management tool accel-config in user space to enable system administrators to configure the device, groups, work queues, and engines, and provide device state and other related information using the Linux sysfs. Specifically, this user space utility has been implemented using json-c/libjson. This lowers the bar for upper-level infrastructure tooling to interface with the kernel and platform hardware, and to represent device configuration information and hierarchical structure.

Fig 1. Intel DSA Linux overview

 

By using the command-line interface provided in the utility, you can list and configure the individual attributes into the device hierarchy and load and save the configuration file through the Linux kernel sysfs read and write. The device hierarchy incorporates device, group, workqueue, and engine. Each device can have multiple work queues and engines, and can assign them into different groups. A workqueue is on-device storage to contain descriptors to be submitted to the device, and an engine is an operational unit in the device to submit descriptors. A portal is a special register through which descriptors are submitted to work queues. The utility utilizes the sysfs interface to solve the implementation problem of configuring multiple values and enabling them individually during bind. With the help of the accel-config, the system administrator can configure work queues for specific use cases, such as “background” tasks where the accelerator is used in parallel with CPU, and performance-critical uses where CPU polls for completion.

Fig 2. Accel-config Utility and Library in Accelerator System

 

libaccel-config is the user-space library wrapping the Intel DSA sysfs ABI. It is developed to get access to work queues by using the mmap portal from the character device and send work to the accelerator using new instruction ENQCMD or MOVDIR64.

This management tool provided by Intel DSA is a novel way to have a user-space utility configure a complex device using the Linux sysfs interface. libaccfg is provided as an API library that provides functions accel-config utilizes and can be linked in by a common application, such as Data Plane Development Kit (DPDK) for use.

Example use cases

Table 1 lists the use case of accel-config and how accel-config cooperates with hardware and the idxd driver.

Use case

idxd driver

/sys/bus/dsa/devices/

accel-config UI

(command line only)

Device init and enumeration

Initialization and enumeration

(how many devices need to be identified)

Walk through to figure out how many devices/groups/wq/engines

available on sysfs

Device info display

read capabilities

(how many devices/wq/engines)

read and display in JSON format

“accel-config list”

Device config save and load

load configuration of device

save current configuration

“accel-config load-config”

”accel-config save-config”

Device attribute configuration

bare-metal configuration

(changes on config file/assign engines/wq to groups)

“accel-config config-wq

<wq attr>”

“accel-config config-engine

< engine attr>”

Device virtualization configuration

virtualization configuration, create new mediated device on guest OS, attach random uuid

“accel-config create-mdev

<wq name>”

Device enable and disable

activate/deactivate device

“accel-config enable-device <device name>”

“accel-config disable-wq

<wq name>”

Table 1. Use case of accel-config

 

Usage of accel-config

After you get the configuration set up, the "save-config" can be used to store the entire configuration. Use "load-config" from system boot-time script to re-initialize each reboot afterwards.

To save the configuration of the enabled device and workqueue, issue the following command and save the config file to a designated location.

sudo accel-config save-config --saved-file=./saved.conf

To load the saved config file:

sudo accel-config load-config -c ./saved.conf

Another option is to configure the device and component individually. In the following example, note that the first digit represents the device number and second digit represents the component number under that device. Thus “group0.0” means group 0 under device 0, and “wq1.0” means wq0 under device 1.


    # setup wq0.0 belongs to group 0 on dsa0
    sudo accel-config config-wq dsa0/wq0.0 --group-id=0
    # setup priority of wq0.0
    sudo accel-config config-wq dsa0/wq0.0 --priority=10
    # setup wq size for wq0.0
    sudo accel-config config-wq dsa0/wq0.0 --wq-size=16
    # setup engine0.0 to belong to group 0
    sudo accel-config config-engine dsa0/engine0.0 --group-id=0
    # setup type of wq0.0
    sudo accel-config config-wq dsa0/wq0.0 --type=kernel
    # setup name of wq0.0
    sudo accel-config config-wq dsa0/wq0.0 --name=”dmaengine”
    # setup mode of wq0.0
    sudo accel-config config-wq dsa0/wq0.0 --mode=shared
    # setup threshold of wq0.0
    sudo accel-config config-wq dsa0/wq0.0 --threshold=15

The workqueue type can be “kernel”, “user”, “mdev”, or “none”. The “kernel” type is used for the kernel use case to request for workqueue. The “user” type is used for regular character device driver use case to request a workqueue. The “mdev” type is used for the virtualization guest to request a workqueue. After setting up the workqueue type, you should assign a workqueue name to it and configure the workqueue mode. The mode can either be “dedicated”, which means the device exposes all the total number of descriptors in one workqueue, or “shared” which means the device reserves the available descriptors among multiple work queues.

For all the configurable attributes under device/workqueue/engine/group, refer to the documentation under source code or go to this GitHub* website for reference: https://github.com/intel/idxd/blob/master/docs/index.md

After completing the configuration of the workqueue and the engine, you can start the device and workqueue using the following command. Note that you need to enable the device first and then the workqueue under that device:


sudo accel-config enable-device dsa0
sudo accel-config enable-wq dsa0/wq0.0
sudo accel-config enable-wq dsa0/wq0.1

To check the device status after configuration and enabling, execute the following command:


    # to list all the components
    sudo accel-config list
    #to list only devices
    sudo accel-config list -D
    #to list only groups
    sudo accel-config list -G
    #to list only workqueues
    sudo accel-config list -Q
    #to list only engines
    sudo accel-config list -E
    #to list device 0
    sudo accel-config list -d dsa0
    #to list device 1
    sudo accel-config list -d dsa1
    #to list group2 in device 1
    sudo accel-config list -g dsa1/group1.2
    #to list wq5 in device 0
    sudo accel-config list -q dsa0/wq0.5

Executing the "list" command without attributes displays the attributes of all components, as shown in the following example. The top level hierarchy starts with the component “device”, and the second-level hierarchy starts with “group” and all its attributes. Fig 3 shows the hierarchy in JSON format.

Fig 3. “accel-config list” output

To set individual attributes, refer to the following examples.


    # config group0.2 use-token-limit
    sudo accel-config config-group dsa0/group0.2 --use-token-limit=1

    #config group-id and wq-size for wq0.3
    sudo accel-config config-wq wq0.3 --group-id=2 --wq-size=5

    #config priority for wq0.2
    sudo accel-config config-wq wq0.2 --priority=3

    #config block-on-default and mode for wq0.4
    sudo accel-config config-wq wq0.4 --block-on-fault=1 --mode=dedicated

    #config engine0.3 group-id as 1
    sudo accel-config config-engine dsa0/engine0.3 --group-id=1

You can check the configured value to see if it has been successfully set.

Fig 4. Set “threshold” attribute and set the value

If you want to disable devices or components to finish up the entire configuration and then save it, execute the following command and note that the workqueue must first be disabled before disabling the device.


    # disable wq0.2
    sudo accel-config disable-wq dsa0/wq0.2
    # disable dsa0
    sudo accel-config disable-device dsa0

This article presents the motivation to develop accel-config, its use cases, and illustrates the way to use it. It is intended to be expanded to incorporate other Intel accelerators with similar exported sysfs interfaces. The library API it offers can be widely used by upper applications to control hardware from user space.

The github repo of accel-config can be accessed at: https://github.com/intel/idxd-config/

The github man page of accel-config can be accessed at: https://github.com/intel/idxd

For any questions regarding using the code, you are welcome to email us  at accel-config@lists.01.org.

References