Sorry, you need to enable JavaScript to visit this website.

Linux Kernel Performance

Linux development evolves rapidly. The performance and scalability of the OS kernel has been a key part of its success. However, discussions have appeared on LKML (Linux Kernel Mailing List) regarding large performance regression between kernel versions. These discussions underscore the need for a systematic and disciplined way to characterize, improve, and test Linux kernel performance. Our goal is to work with the Linux community to further enhance the Linux kernel with consistent performance increases (avoiding degradations) across releases. The information available on this site gives community members better information about what 0-Day and LKP (Linux Kernel Performance) are doing to preserve performance integrity of the kernel.

This 0-Day service is an automated Linux kernel test service that provides comprehensive test coverage of the Linux kernel. It monitors various kernel trees, spanning the mainline tree, the next tree, maintainers’ trees, and key developers’ trees for changes. 0-Day also monitors the Linux Kernel Mail List (LKML) itself. 0-Day performs build, boot, functional, performance, and power tests whenever it detects changes. Our goal is to assist developers to find problems as early as possible so they can be fixed at the first opportunity.

Whenever there are any boot, functional, performance, or power issues detected by the test infrastructure, kernel developers receive email reports from the kbuild test robot. This is a service from 0-Day that automatically reports build failures of Linux kernel code.

0-Day does build tests using more than 100 different kernel configurations. For Intel x86 architecture, static analysis is also performed for selected widely-used configurations. The turnaround time of build results is within hours after a code change is detected. If there is any failure during the build stage, 0-Day will bisect the failure to the first code patch that introduces the failure. That patch author is then notified with the failure information and the steps to reproduce the problem. This allows developers to reproduce the problem in their local environments and to verify their fixes.

Here is an example of the email report produced when there is any build failure:

For kernels that have passed build tests, boot testing is done on x86 bare metal or on virtual machines. For kernels that have passed build tests, more than 80 functional test suites and benchmarks will be run. For each test case or benchmark, there could be tens to hundreds of combinations of setups and parameters. Similar to the build test process, when any regressions are detected, 0-Day bisects the included patchsets to find the first bad commit that introduced the regression. An email report is then be sent to the patch author with the information needed to reproduce the regression. This includes test machine configuration, kernel configuration, dmesg output, test result comparison, and the necessary steps to run the test to reproduce the problem and for the responsible developer to validate the fix.

Here is an example of boot failure report:

This is an example of performance regression report:

All email reports by the 0-Day test infrastructure are archived here:

To learn how to get involves in 0-Day test services, refer to the following link:

Developers can request that their patch gets tested by the 0-Day infrastructure by sending the patch or tree information to the 0-Day mailing list. We’d like to continuously improve our service, so if you have any feedback, please kindly let us know.


Q: What will be reported?

A: 0-Day covers many aspects of the Linux kernel. For the monitored git trees, 0-Day reports build failures, boot failures, functional bugs, and regression/improvement of kernel performance.

Q: Who will receive the report?

A: By default, the regression report is sent to the commit author of the patch that contributed to the regression. By default, 0-Day does not send test successful reports. For boot/function/regression issues, reports are sent to the Linux Kernel Mailing List. If you want to do reports differently from the default method, send your requirements to the 0-Day mailing list.

Q: How to I specify the kconfig to 0-Day?

A: 0-Day tests 100+ different kernel configurations that range from commonly used configurations to random configurations. If you want to request a specific kernel configuration for testing, send an email to the LKML with your config file.

Q: Which git tree and which mailing list will be tested? How can I opt-in or opt-out from it?

A: 0-Day monitors hundreds of git trees and tens of mailing lists. You can obtain detailed tree and mailing list information from the source code under the lkp-tests/repo directory. If you want to add or remove your tree from the 0-Day test system, send an email to the LKML, specifying your git tree URL.

Q: What kernel configurations do 0-Day builds use?

A: 0-Day does allmodconfig and many random config builds.

Q: What test cases and benchmarking tools does 0-Day use for functional, performance, and power testing?

A: 0-Day integrates about 80 functional test suites and benchmarking test cases from open source. Refer to lkp-tests/tests at the following link for test case details:

Q: How does 0-Day analyze the performance data?

A: 0-Day uses key-value pairs to represent performance data and combines the results of multiple runs of a single commit (in a certain environment) to compare to data from previous releases so we can detect performance change.

Q: How does 0-Day track the bug status? How do developers know that their bug is fixed in the new branch HEAD? How does 0-Day send out reminders about old bugs that have not been fixed?

A: 0-Day tracks history status including reported errors/warnings, first bad commits, and other statistics like branch commit IDs to avoid sending duplicate reports to authors.