Sorry, you need to enable JavaScript to visit this website.

Linux Kernel Performance

Linux development evolves rapidly. The performance and scalability of the OS kernel has been a key part of its success. However, discussions have appeared on LKML (Linux Kernel Mailing List) regarding large performance regression between kernel versions. These discussions underscore the need for a systematic and disciplined way to characterize, improve, and test Linux kernel performance. Our goal is to work with the Linux community to further enhance the Linux kernel with consistent performance increases (avoiding degradations) across releases. The information available on this site gives community members better information about what 0-Day and LKP (Linux Kernel Performance) are doing to preserve performance integrity of the kernel.

0-Day CI Linux Kernel Performance Report (v5.6)

BY Rong Chen ON Apr 25, 2020
  1. Introduction

0-Day CI is an automated Linux kernel test service that provides comprehensive test coverage of the Linux kernel. It covers kernel build, static analysis, boot, functional, performance and power tests. This report shows the recent observations of kernel performance status on IA platform based on the test results from 0-Day CI service. It is structured in the following manner:

  • Section 2, merged regressions and improvements in v5.6 release candidates

  • Section 3, test parameter description

  • Section 4, captured regressions and improvements by shift-left testing during developers’ and maintainers’ tree during v5.6 release cycle

  • Section 5, performance comparison among different kernel releases

  • Section 6, test machine list

 

  1. Test parameters

Here are the descriptions for each parameter/field used in the tests. 

 

Classification

Name

Description

General

runtime

Run the test case within a certain time period (seconds or minutes)

 

nr_task

If it is an integer, which means the number of processes/threads (to run the workload) of this job. Default is 1.

If it is a percentage, e.g. 200% means the number of processes/threads is double of cpu number

 

nr_threads

Alias of nr_task

 

iterations

Number to repeat this job

 

test_size

Test disk size or memory size

 

set_nic_irq_affinity

Set NIC interrupt affinity

 

disable_latency_stats

Latency_stats may introduce too much noise if there are too many context switches, allow to disable it

 

transparent_hugepage

Set transparent hugepage policy (/sys/kernel/mm/transparent_hugepage)

 

boot_params:bp1_memmap

Boot parameters of memmap

 

disk:nr_pmem

number of pmem partitions used by test

 

swap:priority

Priority means the  priority  of  the  swap device. priority is a value between -1 and 32767, the default is -1 and higher priority with higher value. 

Test Machine

model

Name of Intel processor microarchitecture

 

brand

Brand name of cpu

 

cpu_number

Number of cpu

 

memory

Size of memory


 

  1. Linux Kernel v5.6 Release Test

The 5.6 release of the Linux kernel was on March 29, 2020. Some of the headline features in this release include Arm EOPD support, time namespaces, the BPF dispatcher and batched BPF map operations (both described in this article), the openat2() system call, the WireGuard virtual private network implementation, the flow queue PIE packet scheduler, nearly complete year-2038 support, many new io_uring features, the pidfd_getfd() system call, the ZoneFS filesystem, the ability to implement TCP congestion-control algorithms in BPF, the dma-buf heaps subsystem, and the removal of the /dev/random blocking pool.

 

0-Day CI monitored the release closely to trace down the performance status on IA platform. 0-Day observed 5 regressions and 6 improvements during feature development phase for v5.6. We will share more detailed information together with correlated patches that led to the results. Note that the assessment is limited by the test coverage 0-Day has now. The list is summarized in the observation summary section.

    1. Observation Summary

0-Day CI observed 5 regressions and 6 improvements during feature development phase for v5.6, which is in the time frame from v5.6-rc1 to v5.6 release. 

Test Indicator

Report

Test Scenario

Test Machine

Development Base

Status

fsmark.files_per_sec

[LKP] [sched/fair] 52262ee567: 15.9% improvement

iterations: 1x

nr_threads: 1t

disk: 1BRD_32G

fs: xfs

fs2: nfsv4

filesize: 4K

test_size: 4G

sync_method: fsyncBeforeClose

nr_files_per_directory: 1fpd

cpufreq_governor: performance

lkp-csl-2sp5

v5.5-rc3


 

merged at at v5.6-rc2

fsmark.files_per_sec

[LKP] [ext4] 244adf6426: 17.5% improvement

iterations: 1x

nr_threads: 64t

disk: 1BRD_48G

fs: ext4

fs2: nfsv4

filesize: 4M

test_size: 40G

sync_method: NoSync

cpufreq_governor: performance

lkp-csl-2ap2

v5.5-rc3

merged at v5.6-rc1

hackbench.throughput

[LKP] [pipe] 0ddad21d3e: 136.1% improvement

nr_threads: 50%

mode: process

ipc: pipe

cpufreq_governor: performance

lkp-csl-2sp7

v5.5

merged at v5.6-rc1

stress-ng.eventfd.ops_per_sec

[LKP] [x86/mm] 763802b53a: 11.1% improvement

nr_threads: 100%

disk: 1HDD

testtime: 1s

class: os

cpufreq_governor: performance

fs: ext4

lkp-csl-2sp5

v5.6-rc6

merged at v5.6-rc7

stress-ng.switch.ops_per_sec

[LKP] [mm] fd4d9c7d0c: -30.5% regression

nr_threads: 100%

disk: 1HDD

testtime: 30s

test: switch

cpufreq_governor: performance

lkp-csl-2sp5

v5.6-rc6

merged at v5.6-rc7, cache align issue

will-it-scale.per_process_ops

[LKP] [perf/x86/amd] 471af006a7: -7.6% regression

nr_task: 16

mode: process

test: mmap1

cpufreq_governor: performance

lkp-bdw-ep6

v5.5-rc3

merged at v5.6-rc1, still in our to-do list 

will-it-scale.per_process_ops

[LKP] [signal] fda31c5029: 58.4% improvement

nr_task: 50%

mode: process

test: signal1

cpufreq_governor: performance

lkp-csl-2ap3

v5.6-rc3

merged at v5.6-rc4

will-it-scale.per_process_ops

[LKP] [y2038] 412c53a680: 11.7% improvement

nr_task: 100%

mode: process

test: mmap2

cpufreq_governor: performance

lkp-skl-2sp7

v5.6-rc2

merged at v5.6-rc3

will-it-scale.per_process_ops

[LKP] [futex] 8019ad13ef: -97.8% regression

nr_task: 100%

mode: process

test: futex2

cpufreq_governor: performance

lkp-skl-fpga01

v5.6-rc4

merged at v5.6-rc6, the regression has been fixed in v5.6-rc6.

will-it-scale.per_process_ops

[LKP] [locks] 6d390e4b5d: -96.6% regression

nr_task: 100%

mode: process

test: lock1

cpufreq_governor: performance

lkp-knm01

v5.6-rc4

merged at v5.6-rc5, the regression has been fixed in v5.6-rc7.

will-it-scale.per_thread_ops

[LKP] [xdp] 332f22a60e: -11.4% regression

nr_task: 100%

mode: thread

test: dup1

cpufreq_governor: performance

lkp-knm01

v5.5-rc1

merged at v5.6-rc1, but the regression has disappeared in v5.6

    1. hackbench.throughput

Hackbench is both a benchmark and a stress test for the Linux kernel scheduler. It's main job is to create a specified number of pairs of schedulable entities (either  threads or traditional processes) which communicate via either sockets or pipes and time how long it takes for each pair to send data back and forth.

      1. scenario: process pipe

 

Commit 0ddad21d3e was reported to have 136.1% of hackbench.throughput when comparing to v5.5. It was merged to the mainline at v5.6-rc1.

 

Correlated commits

0ddad21d3e

pipe: use exclusive waits when reading or writing

branch

linux/master

report

[LKP] [pipe] 0ddad21d3e: 136.1% improvement

test scenario

nr_threads: 100%

nr_threads: 50%

mode: process

ipc: pipe

cpufreq_governor: performance

test machine

lkp-csl-2sp7

status

merged at v5.6-rc1

 

    1. stress-ng.switch.ops_per_sec

stress-ng will stress test a computer system in various selectable ways. It was designed to exercise various physical subsystems of a computer as well as the various operating system kernel interfaces.

 

    1. scenario: 100%-1HDD-30s-switch

 

Commit fd4d9c7d0c was reported to have -30.5% of stress-ng.switch.ops_per_sec when comparing to v5.6-rc6. It was merged to the mainline at v5.6-rc7.

 

Correlated commits

fd4d9c7d0c

mm: slub: add missing TID bump in kmem_cache_alloc_bulk()

branch

linus/master

report

[LKP] [mm] fd4d9c7d0c: -30.5% regression

test scenario

nr_threads: 100%

disk: 1HDD

testtime: 30s

test: switch

cpufreq_governor: performance

test machine

lkp-csl-2sp5

status

merged at v5.6-rc7, cache align issue

 

  1. will-it-scale.per_process_ops

Will-it-scale takes a test case and runs it from 1 through to n parallel copies to see if the test case will scale. It builds both process and threads based tests in order to see any differences between the two.

    1. Scenario: process lock1

 

 

Commit 6d390e4b5d was reported to have -96.6% of will-it-scale.per_process_ops when comparing to v5.6-rc4. It was merged to the mainline at v5.6-rc5.

 

Correlated commits

6d390e4b5d

locks: fix a potential use-after-free problem when wakeup a waiter

branch

linus/master

report

[LKP] [locks] 6d390e4b5d: -96.6% regression

test scenario

nr_task: 100%

mode: process

test: lock1

cpufreq_governor: performance

test machine

lkp-knm01

status

merged at v5.6-rc5, the regression has been fixed in v5.6-rc7.


 

  1. Shift-Left Testing

Beyond testing trees in the upstream kernel, 0-Day CI also tests developers’ and maintainers’ trees, which can catch issues earlier and reduce wider impact. We call it “shift-left” testing. During the v5.6 release cycle, 0-Day CI had reported 24 major performance regressions and 19 major improvements by doing shift-left testing. We will share more detailed information together with possible code changes that led to this result for some of these, though the assessment is limited by the test coverage we have now. The whole list is summarized in the report summary section.

    1. Report Summary

0-Day CI had reported 24 performance regressions and 19 improvements by doing shift-left testing on developer and maintainer repos.

 

Test Indicator

Mail

Test Scenario

Test Machine

Status

aim7.jobs-per-min

[LKP] [xfs] 41d44159a3: -23.2% regression

disk: 1BRD_48G

fs: xfs

test: sync_disk_rw

load: 600

cpufreq_governor: performance

lkp-skl-2sp7

currently not merged, no response from author yet

aim7.jobs-per-min

[LKP] [x86/unwind/orc] 25a14cb54d: -6.7% regression

disk: 1BRD_48G

fs: xfs

test: disk_wrt

load: 3000

cpufreq_governor: performance

lkp-skl-2sp7

currently not merged, no response from author yet

aim7.jobs-per-min

[LKP] [locking/qspinlock] 372cdd28b7: 76.7% improvement

disk: 4BRD_12G

md: RAID0

fs: btrfs

test: disk_rr

load: 500

cpufreq_governor: performance

lkp-csl-2ap2

currently not merged

aim7.jobs-per-min

[LKP] [xfs] 542951592c: 22.9% improvement

disk: 4BRD_12G

md: RAID1

fs: xfs

test: disk_cp

load: 3000

cpufreq_governor: performance

lkp-csl-2ap2

currently not merged

aim7.jobs-per-min

[LKP] [btrfs] c75e839414: -8.9% regression

disk: 4BRD_12G

md: RAID0

fs: btrfs

test: disk_wrt

load: 1500

cpufreq_governor: performance

lkp-csl-2ap2

currently not merged, no response from author yet

filebench.sum_bytes_mb/s

[LKP] [xfs] 22d05173fc: -47.5% regression

disk: 1HDD

fs: xfs

test: fileserver.f

cpufreq_governor: performance

lkp-hsw-d01

currently not merged, no response from author yet

filebench.sum_bytes_mb/s

[LKP] [f2fs] f9c20981ec: -61.0% regression

disk: 1HDD

fs: f2fs

test: webproxy.f

cpufreq_governor: performance

lkp-hsw-d01

currently not merged, no response from author yet

fio.read_bw_MBps

[LKP] [mm/workingset] 323c95f095: 19.5% improvement

disk: 2pmem

fs: ext4

runtime: 200s

nr_task: 50%

time_based: tb

rw: randrw

bs: 2M

ioengine: mmap

test_size: 200G

cpufreq_governor: performance

lkp-csl-2sp6

currently not merged

fio.read_bw_MBps

[LKP] c9d32d8e86: 94.9% improvement

disk: 2pmem

fs: xfs

runtime: 200s

nr_task: 50%

time_based: tb

rw: randrw

bs: 4k

ioengine: sync

test_size: 200G

cpufreq_governor: performance

lkp-csl-2sp6

currently not merged

fio.write_bw_MBps

[LKP] [btrfs] d0dfc759a9: 22.9% improvement

disk: 2pmem

fs: btrfs

runtime: 200s

nr_task: 50%

time_based: tb

rw: randwrite

bs: 4k

ioengine: libaio

test_size: 100G

cpufreq_governor: performance

lkp-hsw-ep4

currently not merged

fio.write_bw_MBps

[LKP] [btrfs] 4b9197583d: 9.1% improvement

disk: 2pmem

fs: btrfs

runtime: 200s

nr_task: 50%

time_based: tb

rw: randwrite

bs: 4k

ioengine: mmap

test_size: 100G

cpufreq_governor: performance

lkp-csl-2sp6

currently not merged

fio.write_bw_MBps

[LKP] [btrfs] 1eb52c8bd8: -46.1% regression

runtime: 300s

disk: 1SSD

fs: btrfs

nr_task: 100%

test_size: 128G

rw: write

bs: 4k

ioengine: sync

direct: direct

cpufreq_governor: performance

lkp-bdw-ex2

currently not merged, no response from author yet

fio.write_bw_MBps

[LKP] 567fcac239: -7.8% regression

runtime: 300s

disk: 1SSD

fs: ext4

nr_task: 100%

test_size: 128G

rw: write

bs: 4k

ioengine: sync

direct: direct

cpufreq_governor: performance

lkp-bdw-ex2

currently not merged, no response from author yet

fsmark.files_per_sec

[LKP] [sched/fair] 01ef59fa82: 17.4% improvement

iterations: 1x

nr_threads: 1t

disk: 1BRD_32G

fs: xfs

fs2: nfsv4

filesize: 4K

test_size: 4G

sync_method: fsyncBeforeClose

nr_files_per_directory: 1fpd

cpufreq_governor: performance

lkp-csl-2sp5

currently not merged

fwq.fwq.med

[LKP] [cpufreq] 909c0e9cc1: 210.0% improvement

nr_task: 100%

samples: 100000ss

iterations: 18x

cpufreq_governor: powersave

lkp-bdw-de1

currently not merged

hackbench.throughput

[LKP] [sched/fair] 15e7470dfc: 11.2% improvement

nr_threads: 100%

mode: threads

ipc: pipe

cpufreq_governor: performance

lkp-skl-fpga01

currently not merged

hackbench.throughput

[LKP] 9c921492e9: -33.4% regression

runtime: 20s

nr_threads: 50%

mode: process

ipc: pipe

cpufreq_governor: performance

lkp-csl-2sp7

currently not merged, no response from author yet

netperf.Throughput_Mbps

[LKP] [kernel] 2a8b6290ee: 12.6% improvement

ip: ipv4

runtime: 300s

nr_threads: 25%

cluster: cs-localhost

send_size: 10K

test: SCTP_STREAM_MANY

cpufreq_governor: performance

lkp-skl-2sp7

currently not merged

netperf.Throughput_Mbps

[LKP] [sched/fair] 59901cb452: -27.3% regression

ip: ipv4

runtime: 300s

nr_threads: 25%

cluster: cs-localhost

send_size: 5K

test: TCP_SENDFILE

cpufreq_governor: performance

lkp-skl-2sp7

currently not merged, no response from author yet

netperf.Throughput_total_tps

[LKP] [sched/fair] 34afb69d86: -4.7% regression

ip: ipv4

runtime: 30s

nr_threads: 50%

cluster: cs-localhost

test: TCP_RR

cpufreq_governor: performance

lkp-csl-2sp7

currently not merged, no response from author yet

phoronix-test-suite.aom-av1.0.frames_per_second

[LKP] [sched/numa] f6183ef98b: -25.0% regression

test: aom-av1-1.2.0

cpufreq_governor: performance

lkp-nhm-2ep1

currently not merged, no response from author yet

reaim.jobs_per_min

[LKP] [sched/fair] 070f5e860e: -10.5% regression

runtime: 300s

nr_task: 100%

test: five_sec

cpufreq_governor: performance

lkp-ivb-d04

currently not merged, no response from author yet

stress-ng.fiemap.ops_per_sec

[LKP] [ext4] d3b6f23f71: -60.5% regression

nr_threads: 10%

disk: 1HDD

testtime: 1s

class: os

cpufreq_governor: performance

fs: ext4

lkp-csl-2sp5

currently not merged, the author is working on it.

stress-ng.hdd.ops_per_sec

[LKP] [mm/lru] 2f14ef7251: -69.1% regression

nr_threads: 100%

disk: 1HDD

testtime: 1s

class: io

cpufreq_governor: performance

lkp-csl-2sp5

currently not merged, no response from author yet

stress-ng.hdd.ops_per_sec

[LKP] [mm/lru] 2e654b6306: -69.5% regression

nr_threads: 100%

disk: 1HDD

testtime: 30s

class: io

cpufreq_governor: performance

lkp-csl-2sp5

currently not merged, the author is working on it.

stress-ng.hdd.ops_per_sec

[LKP] [mm/lru] 7e653afc0d: -69.4% regression

nr_threads: 100%

disk: 1HDD

testtime: 30s

class: io

cpufreq_governor: performance

lkp-csl-2sp5

currently not merged, no response from author yet

stress-ng.msg.ops_per_sec

[LKP] [nfs] c5654df66d: 15.5% improvement

nr_threads: 100%

disk: 1HDD

testtime: 1s

class: scheduler

cpufreq_governor: performance

sc_pid_max: 4194304

lkp-csl-2sp5

currently not merged

stress-ng.symlink.ops_per_sec

[LKP] [ext4] dcdc8066b4: 60.7% improvement

nr_threads: 10%

disk: 1HDD

testtime: 1s

class: os

cpufreq_governor: performance

fs: ext4

lkp-csl-2sp5

currently not merged

unixbench.score

[LKP] [locking/qspinlock] 327c3f5fc8: -24.2% regression

runtime: 300s

nr_task: 30%

test: context1

cpufreq_governor: performance

lkp-bdw-ex2

currently not merged, no response from author yet

vm-scalability.median

[LKP] [mm/vmscan] dcf33bfdfb: 402.4% improvement

runtime: 300

thp_enabled: never

thp_defrag: always

nr_task: 32

nr_pmem: 1

test: swap-w-rand-mt

bp_memmap: 96G!18G

cpufreq_governor: performance

lkp-csl-2sp6

currently not merged

vm-scalability.median

[LKP] [sched/fair] c32b430829: 5.8% improvement

runtime: 300s

size: 8T

test: anon-w-seq

cpufreq_governor: performance

lkp-csl-2ap4

currently not merged

vm-scalability.median

[LKP] [thp] db001b7115: 8.9% improvement

runtime: 300s

size: 8T

test: anon-cow-seq

cpufreq_governor: performance

lkp-cfl-e1

currently not merged

vm-scalability.median

[LKP] 5fba1f38eb: -10.6% regression

runtime: 300

thp_enabled: never

thp_defrag: always

nr_task: 8

nr_pmem: 1

test: swap-w-rand-mt

bp_memmap: 96G!18G

cpufreq_governor: performance

lkp-csl-2sp6

currently not merged, no response from author yet

will-it-scale.per_process_ops

[LKP] [kernel] 247f5d7caa: 9.6% improvement

nr_task: 16

mode: process

test: malloc1

cpufreq_governor: performance

lkp-csl-2ap2

currently not merged

will-it-scale.per_process_ops

[LKP] 361fdccf21: 36.3% improvement

nr_task: 100%

mode: process

test: getppid1

cpufreq_governor: performance

lkp-knm01

currently not merged

will-it-scale.per_process_ops

[LKP] [sched] db8e976e4a: 15.8% improvement

nr_task: 100%

mode: process

test: mmap1

cpufreq_governor: performance

lkp-knm01

currently not merged

will-it-scale.per_process_ops

[LKP] [mm/lru] 86573dbf59: -64.0% regression

nr_task: 100%

mode: process

test: page_fault2

cpufreq_governor: performance

lkp-skl-2sp7

currently not merged, no response from author yet

will-it-scale.per_process_ops

[LKP] [cpufreq] 06c4d00466: -53.4% regression

nr_task: 16

mode: process

test: read2

cpufreq_governor: performance

lkp-ivb-d02

currently not merged, no response from author yet

will-it-scale.per_process_ops

[LKP] [x86/entry] 8c5045f89b: -12.9% regression

nr_task: 100%

mode: process

test: lseek1

cpufreq_governor: performance

lkp-knm01

currently not merged, no response from author yet

will-it-scale.per_process_ops

[LKP] [mm/lru] 96f4b4633c: -64.2% regression

nr_task: 100%

mode: process

test: page_fault2

cpufreq_governor: performance

lkp-skl-2sp7

currently not merged, no response from author yet

will-it-scale.per_process_ops

[LKP] [signal] fda31c5029: 58.4% improvement

nr_task: 50%

mode: process

test: signal1

cpufreq_governor: performance

lkp-csl-2ap3

currently not merged

will-it-scale.per_thread_ops

[LKP] [rcu] 76550daa4d: -10.5% regression

nr_task: 100%

mode: thread

test: open2

cpufreq_governor: performance

lkp-knm01

currently not merged, no response from author yet

will-it-scale.per_thread_ops

[LKP] [futex] 222993395e: 1507.5% improvement

nr_task: 16

mode: thread

test: futex1

cpufreq_governor: performance

lkp-csl-2ap2

currently not merged

    1. aim7.jobs-per-min

aim7 is a traditional UNIX system level benchmark suite which is used to test and measure the performance of a multiuser system. 

 

      1. scenario: disk_rr test on btrfs

 

Commit 372cdd28b7 was reported to have 76.7% of aim7.jobs-per-min when comparing to v5.6-rc6.

 

Correlated commits

372cdd28b7

locking/qspinlock: Introduce CNA into the slow path of qspinlock

branch

peterz-queue/locking/wip-cna

report

[LKP] [locking/qspinlock] 372cdd28b7: 76.7% improvement

test scenario

disk: 4BRD_12G

md: RAID0

fs: btrfs

test: disk_rr

load: 500

cpufreq_governor: performance

test machine

lkp-csl-2ap2

status

currently not merged

 

    1. filebench.sum_bytes_mb/s

Filebench is a file system and storage benchmark that can generate a large variety of workloads. Unlike typical benchmarks it is extremely flexible and allows to specify application's I/O behavior using its extensive Workload Model Language (WML). Users can either describe desired workloads from scratch or use (with or without modifications) workload personalities shipped with Filebench (e.g., mail-, web-, file-, and database-server workloads). Filebench is equally good for micro- and macro-benchmarking, quick to set up, and relatively easy to use.

      1. scenario: webproxy.f test

 

 

Commit f9c20981ec was reported to have -61.0% of filebench.sum_bytes_mb/s when comparing to v5.6-rc4.

 

Correlated commits

f9c20981ec

f2fs: Handle casefolding with Encryption

branch

fscrypt/fscrypt-casefold

report

[LKP] [f2fs] f9c20981ec: -61.0% regression

test scenario

disk: 1HDD

fs: f2fs

test: webproxy.f

cpufreq_governor: performance

test machine

lkp-hsw-d01

status

currently not merged, no response from author yet

 

    1. fio.write_bw_MBps

Fio is a tool that will spawn a number of threads or processes doing a particular type of I/O action as specified by the user.

    1. scenario: write on btrfs

 

Commit 1eb52c8bd8 was reported to have -46.1% of fio.write_bw_MBps when comparing to v5.6-rc7.

 

Correlated commits

1eb52c8bd8

btrfs: get rid of one layer of bios in direct I/O

branch

linux-next/master

report

[LKP] [btrfs] 1eb52c8bd8: -46.1% regression

Test Scenario

runtime: 300s

disk: 1SSD

fs: btrfs

nr_task: 100%

test_size: 128G

rw: write

bs: 4k

ioengine: sync

direct: direct

cpufreq_governor: performance

Test Machine

lkp-bdw-ex2

status

currently not merged, no response from author yet

 

  1. will-it-scale.per_thread_ops

Will-it-scale takes a test case and runs it from 1 through to n parallel copies to see if the test case will scale. It builds both process and threads based tests in order to see any differences between the two.

    1. Scenario: thread dup1

 

 

Commit 361fdccf21 was reported to have 36.3% of will-it-scale.per_process_ops when comparing to v5.6-rc2.

 

Correlated commits

361fdccf21

x86/mm: Thread pgprot_t through init_memory_mapping()

branch

p2pmem/remap_pages_cache_v3

report

[LKP] 361fdccf21: 36.3% improvement

test scenario

nr_task: 100%

mode: process

test: getppid1

cpufreq_governor: performance

test machine

lkp-knm01

status

currently not merged

 

  1. Latest Release Performance Comparing

 

This session gives some information about the performance difference among different kernel releases, especially between v5.6 and v5.5. There are 50+ performance benchmarks running in 0-Day CI, and we selected 9 benchmarks which historically showed the most regressions/improvements reported by 0-Day CI. Some typical configuration/parameters are used to run the test. For some of the regressions from the comparison, 0-Day did not successfully bisect it thus no related report sent out during the release development period, but it is still worth checking. The root cause to cause the regressions won’t be covered in this session. 

 

In the following figures, the value on the Y-axis is the relative performance number. We used the v5.5 data as the base (performance number is 100).

    1. test suite: vm-scalability

vm-scalability exercises functions and regions of the mm subsystem of the Linux kernel. Below 4 tests show the typical test results. 

 

vm-scalability Test 1

vm-scalability Test 2

 

Here are the test configuration and performance test summary for above tests:                                                                                                                                                                                                           

 

vm-scalability Test 1 

vm-scalability Test 2

test machine

model: Broadwell-EX

brand: Intel(R) Xeon(R) CPU E7-8890 v4 @ 2.20GHz

cpu_number: 160

memory: 256G

model: Skylake

cpu_number: 104

memory: 192G

runtime

300s

300s

size

1T

No requirement

vm-scalability test parameter

test case: lru-shm

test case: small-allocs

performance summary

vm-scalability.throughput on kernel v5.6 has -8.68% regression when comparing to v5.5

vm-scalability.throughput on kernel v5.6 has 9.83% improvement when comparing to v5.5

 

    1. test suite: will-it-scale

Will-it-scale takes a test case and runs it from 1 through to n parallel copies to see if the test case will scale. It builds both process and threads based tests in order to see any differences between the two.

 

will-it-scale  Test 1

Will-it-scale Test 2

will-it-scale  Test 3

will-it-scale  Test 4

 

Here are the parameters and performance test summary for above tests:                                                                                                                                                                                                                          

 

will-it-scale Test 1 

will-it-scale Test 2 

will-it-scale Test 3 

will-it-scale Test 4 

test machine

model: Broadwell-EX

brand: Intel(R) Xeon(R) CPU E7-8890 v4 @ 2.20GHz

cpu_number: 160

memory: 256G

model: Cascade Lake

brand: Intel(R) Xeon(R) Platinum 9242 CPU @ 2.30GHz

cpu_number: 192

memory: 192G

model: Knights Mill

brand: Intel(R) Xeon Phi(TM) CPU 7295 @ 1.50GHz

cpu_number: 288

memory: 80G

model: Knights Mill

brand: Intel(R) Xeon Phi(TM) CPU 7295 @ 1.50GHz

cpu_number: 288

memory: 80G

nr_task

No requirement

16

100%

100%

will-it-scale test parameter

test case: open1

mode: process

test case: pread3

mode: process

test case: unlink1

mode: process

test case: signal1

summary

will-it-scale.per_process_ops on kernel v5.6 has -68.73% regression when comparing to v5.5

will-it-scale.per_process_ops on kernel v5.6 has -3.12% regression when comparing to v5.5

will-it-scale.per_process_ops on kernel v5.6 has 34.85% improvement when comparing to v5.5

will-it-scale.per_process_ops on kernel v5.6 has 594.15% improvement when comparing to v5.5

 

    1. test suite: unixbench

UnixBench is a system benchmark to provide a basic indicator of the performance of a Unix-like system.

 

Unixbench Test 1

Unixbench Test 2

 

Here are the test configuration and performance test summary for above tests:                                                                                                                                                                                                           

 

Unixbench Test 1 

Unixbench Test 2

test machine

model: Cascade Lake

brand: Intel(R) Xeon(R) Platinum 9242 CPU @ 2.30GHz

cpu_number: 192

memory: 192G

model: Ivy Bridge

brand: Intel(R) Core(TM) i7-3770K CPU @ 3.50GHz

cpu_number: 8

memory: 16G

runtime

300s

300s

nr_task

30%

30%

unixbench test parameter

test case: fsbuffer

test case: shell8

performance summary

unixbench.score on kernel v5.6 is almost the same as that in v5.5

unixbench.score on kernel v5.6 has 4.05% improvement when comparing to v5.5

 

    1. test suite: reaim

reaim updates and improves the existing Open Source AIM 7 benchmark. aim7 is a traditional UNIX system level benchmark suite which is used to test and measure the performance of a multiuser system.

 

reaim Test 1                             

reaim Test 2

Here are the test configuration and performance test summary for above tests:                                                                                                                                                                                                                      

 

reaim Test 1 

eaim Test 2

test machine

model: Haswell-EP

brand: Intel(R) Xeon(R) CPU E5-2699 v3 @ 2.30GHz

cpu_number: 72

memory: 256G

model: Cascade Lake

brand: Intel(R) Xeon(R) Gold 6252 CPU @ 2.10GHz

cpu_number: 96

memory: 256G

runtime

300s

300s

nr_task

100%

100

disk

No requirement

1HDD

fs

No requirement

btrfs

reaim test parameter

test case: custom

nr_job: 3000

test: disk

performance  summary

reaim.jobs_per_min on kernel v5.6 has -12.61% regression when comparing to v5.5

reaim.jobs_per_min on kernel v5.6 has 62.33% improvement when comparing to v5.5

                                                                                                                                                                                                     

    1. test suite: pigz

pigz, which stands for Parallel Implementation of GZip, is a fully functional replacement for gzip that exploits multiple processors and multiple cores to the hilt when compressing data.

pigz Test 1 


 

 

Here are the test configuration and performance test summary for above tests:          

 

 

pigz Test 1

test machine

model: Broadwell-DE

brand: Intel(R) Xeon(R) CPU D-1541 @ 2.10GHz

cpu_number: 16

memory: 48G

nr_threads

100%

pigz Test parameter

blocksize: 128K

performance  summary

pigz.throughput on kernel v5.6 is almost the same as that in v5.5

            

  1. test suite: netperf

Netperf is a benchmark that can be used to measure the performance of many different types of networking. It provides tests for both unidirectional throughput, and end-to-end latency.

 

netperf Test 1

netperf Test 2

Here are the test configuration and performance test summary for above tests:                                                                                                                                                                                                                          

 

netperf Test 1 

netperf Test 2 

test machine

model: Coffee Lake

brand: Intel(R) Xeon(R) E-2278G CPU @ 3.40GHz

cpu_number: 16

memory: 32G

model: Coffee Lake

brand: Intel(R) Xeon(R) E-2278G CPU @ 3.40GHz

cpu_number: 16

memory: 32G

disable_latency_stats

1

1

set_nic_irq_affinity

1

1

runtime

300s

300s

nr_threads

25%

50%

ip

ipv4

ipv4

netperf test parameter

test case: SCTP_STREAM_MANY

send_size: 10K

test case: UDP_RR

performance  summary

netperf.Throughput_total_Mbps on kernel v5.6 has -3.79% regression when comparing to v5.5

netperf.Throughput_tps on kernel v5.6 has 9.43% improvement when comparing to v5.5

 

    1. test suite: hackbench

Hackbench is both a benchmark and a stress test for the Linux kernel scheduler. It's  main job  is  to  create a specified number of pairs of schedulable entities (either threads or traditional processes) which communicate via either sockets or pipes and time how long  it takes for each pair to send data back and forth.

hackbench Test 1

hackbench Test 2

hackbench Test 3

hackbench Test 4

Here are the test configuration and performance test summary for above tests:                                                                                                                                                                                                           

 

hackbench Test 1 

hackbench Test 2 

hackbench Test 3 

hackbench Test 4 

test machine

model: Haswell-EP

brand: Intel(R) Xeon(R) CPU E5-2699 v3 @ 2.30GHz

cpu_number: 72

memory: 256G

model: Haswell-EP

brand: Intel(R) Xeon(R) CPU E5-2699 v3 @ 2.30GHz

cpu_number: 72

memory: 256G

model: Coffee Lake

brand: Intel(R) Xeon(R) E-2278G CPU @ 3.40GHz

cpu_number: 16

memory: 32G

model: Cascade Lake

brand: Intel(R) Xeon(R) Gold 6252 CPU @ 2.10GHz

cpu_number: 96

memory: 256G

runtime

No requirement

No requirement

No requirement

20s

disable_latency_stats

1

1

1

1

nr_task

1600%

1600%

100%

50%

unixbench test parameter

iterations: 18

mode: process

ipc: pipe

iterations: 18

mode: threads

ipc: socket

mode: process

ipc: pipe

mode: process

ipc: pipe

performance summary

hackbench.throughput on kernel v5.6 has -11.72% regression when comparing to v5.5

hackbench.throughput on kernel v5.6 has -6.15% regression when comparing to v5.5

hackbench.throughput on kernel v5.6 has 92.3% improvement when comparing to v5.5

hackbench.throughput on kernel v5.6 has 153.32% improvement when comparing to v5.5

            

    1. test suite: fio

Fio was originally written to save me the hassle of writing special test case programs when I wanted to test a specific workload, either for performance reasons or to find/reproduce a bug.

fio Test 1 

  

fio Test 2

Here are the test configuration and performance test summary for above tests:                                                                                                                                                                                                                          

 

fio Test 1

fio Test 2

test machine

model: Ivy Bridge-EP

brand: Intel(R) Xeon(R) CPU E5-2697 v2 @ 2.70GHz

cpu_number: 48

memory: 64G

model: Haswell-EP

brand: Intel(R) Xeon(R) CPU E5-2699 v3 @ 2.30GHz

cpu_number: 72

memory: 256G

runtime

20m

300s

file system

ext4

ext4

disk

1HDD

1SSD

test_size

32G

No requirement

nr_task

100%

64

fio test parameter

ffio-setup-basic:

  rw: randwrite

  bs: 4k

  ioengine: sync

fio-setup-basic:

  rw: randwrite

  bs: 4k

  ioengine: sync

  test_size: 512g

performance  summary

fio.write_iops on kernel v5.6 has -15.99% regression when comparing to v5.5

fio.write_bw_MBps on kernel v5.6 has 16.73% improvement when comparing to v5.5

 

    1. test suite: ebizzy

ebizzy is designed to generate a workload resembling common web application server workloads. It is highly threaded, has a large in-memory working set, and allocates and deallocates memory frequently.

ebizzy Test 1

 

 

Here are the test configuration and performance test summary for above test:                                                                                                                                                                                                                          

 

ebizzy Test 1

test machine

model: Haswell

brand: Intel(R) Core(TM) i7-4770 CPU @ 3.40GHz

cpu_number: 8

memory: 8G

nr_threads

200%

iterations

100x

ebizzy test parameter

duration: 10s

performance  summary

ebizzy.throughput on kernel v5.6 is almost the same as that in v5.5


 

  1. Test Machines

    1. IVB Desktop

model

Ivy Bridge

brand

Intel(R) Core(TM) i3-3220 CPU @ 3.30GHz

cpu number

8

memory

16G

 

model

Ivy Bridge

brand

Intel(R) Core(TM) i3-3220 CPU @ 3.30GHz

cpu number

4

memory

8G

 

    1. SKL SP

model

Skylake

brand

Intel(R) Xeon(R) CPU E5-2697 v2 @ 2.70GHz

cpu number

80

memory

64G

 

    1. BDW EP

model

Broadwell-EP

brand

Intel(R) Xeon(R) CPU E5-2699 v4 @ 2.20GHz

cpu number

88

memory

128G

 

    1. HSW EP

model

Haswell-EP

brand

Intel(R) Xeon(R) CPU E5-2699 v3 @ 2.30GHz

cpu number

72

memory

128G

 

    1. IVB EP

model

Ivy Bridge-EP

brand

Intel(R) Xeon(R) CPU E5-2690 v2 @ 3.00GHz

cpu number

40

memory

384G

 

model

Ivytown Ivy Bridge-EP

brand

Intel(R) Xeon(R) CPU E5-2697 v2 @ 2.70GHz

cpu number

48

memory

64G

 

    1. HSX EX

model

Brickland Haswell-EX

brand

Intel(R) Xeon(R) CPU E7-8890 v3 @ 2.50GHz

cpu number

144

memory

512G