Sorry, you need to enable JavaScript to visit this website.

Feedback

Your feedback is important to keep improving our website and offer you a more reliable experience.

Solving VM Live Migration Failure using CPU Feature Analysis

BY Tony Su, Feng Shaohe ON Sep 30, 2019

This article describes the solution for a Virtual Machine (VM) migration failure, caused by a CPU’s inability to migrate from an older generation to a newer generation on an Intel enterprise platform. We conducted a deep-dive investigation to find the root cause as well as a solution to this issue, so this article can be used as a reference for other cloud OEMs and vendors with a similar situation.

Issue Reported

In July 2019, we received an issue report from a cloud customer stating their live migration of VM failed to transfer data from the Intel® Xeon® Processor E5-26xx platform (formerly Sandy Bridge) platform to the Intel® Xeon® Processor E5-26xx v4 IBRS platform (formerly Broadwell). After executing trial-and-error experiments, the customer requested Intel technical support complete a deep investigation and find a solution.

Issue Details

The customer had a batch of CPUs based on Sandy Bridge running OpenStack* with libvirt and VMs. Their newly purchased CPUs are based on Broadwell. When they tried to do VM live migration from Sandy Bridge to Broadwell, OpenStack reported “incompatible CPU” and refused to execute migration.

The error log is shown below:

Unacceptable CPU info: CPU doesn't have compatibility.
ERROR nova.virt.libvirt.driver [-] [instance: 7879eec3-9b60-4fb4-beff-a2a895cf0e0f] Live Migration failure: internal error Unknown CPU feature monitor

This error message confused the customer, they said, because they expect backward compatibility among generations and product family.

Issue Investigation

Customer Experiments

After the customer noticed the ‘Unknown CPU feature monitor’ error message, they checked the difference between CPU feature lists of Sandy Bridge and Broadwell platform, and confirmed the ‘monitor’ feature shows ‘missing / disappeared’ in Broadwell.

Sandy Bridge CPU Feature List

$ nova hypervisor-show
...
| cpu_info | {"vendor": "Intel", "model": "SandyBridge", "arch": "x86_64", "features": ["invtsc", "invpcid", "erms", "bmi2", "smep", "avx2", "bmi1", "f
sgsbase", "abm", "pdpe1gb", "rdrand", "f16c", "osxsave", "movbe", "dca", "pcid", "pdcm", "xtpr", "fma", "tm2", "est", "smx", "vmx", "ds_cpl", monitor, "dtes64",
"pbe", "tm", "ht", "ss", "acpi", "ds", "vme"], "topology": {"cores": 8, "threads": 2, "sockets": 1}} |

Broadwell-IBRS CPU Feature List

Note:  Broadwell is the CPU model; Broadwell-IBRS (indirect branch restricted speculation) is a KVM CPU configuration used to mediate the Spectrum and Meltdown issues. In this case, we can regard Broadwell-IBRS as an equivalent to Broadwell.

$ nova hypervisor-show
...
| cpu_info             | {"vendor": "Intel", "model": "Broadwell-IBRS", "arch": "x86_64", "features": ["invtsc", "abm", "pdpe1gb", "ssbd", "stibp", "rdrand", "f16c
", "osxsave", "dca", "pdcm", "xtpr", "tm2", "est", "smx", "vmx", "ds_cpl", "dtes64", "pbe", "tm", "ht", "ss", "acpi", "ds", "vme"], "topology": {"cores": 8, "threa
ds": 2, "sockets": 1}} |

The customer tried to modify the OpenStack source code and comment out the CPU feature compatibility check. The customer also tried to modify the libvirt cpu_map and comment out monitor-related items. Using this dirty workaround, they were able to complete the migration, however, to avoid the chance of related issues in the future, they asked Intel technical experts for further help.

Intel’s Investigation

The Missing Monitor Feature

When we learned the monitor feature was missing in the newer generation CPU, we decided to investigate the complete history of the monitor feature. We learned the monitor feature was actually two x86 instructions: monitor and “MWAIT”. These instructions were introduced in the SSE3 (Streaming SIMD Extensions 3) instruction set in early 2004 with the Prescott revision of the Pentium(r) 4 processor, and were used to optimize multi-threaded applications, giving processors with Hyper-Threading better performance.

According to Intel spec Prescott New Instructions Software Developer’s Guide (Doc#: 252490-004 January 2004), a programmer must first check the availability of the ‘MWAIT/MONITOR’ instruction using CPUID instructions before actually using it. This check indicates these two instructions might be removed or disabled in future CPU generations - potentially for better performance or power efficiency.

Prescott New Instructions Software Developer’s Guide (Doc#: 252490-004 January 2004)
     The extended feature flag bit 3 [CPUID Function 01, ECX:3] indicates availability for the MONITOR/MWAIT instruction at ring 0 and conditional availability at ring level 1 through 3.

Intel provided a method for the operating system or system BIOS to disable/enable the instructions using IA32_MISC_ENABLE MSR.

IA32_MISC_ENABLE MSR (Addr: 0x1A0H)
[Bit 18] (bit index started from 0)
    Ø  When this bit is set to 0, the MONITOR feature flag is not set (CPUID.01H:ECX[bit 3] = 0).
    This indicates that MONITOR/MWAIT is not supported.
    Ø  When this bit is set to 1 (default), MONITOR/MWAIT is supported CPUID.01H:ECX[bit 3] = 1).

In addition, the reference BIOS offers a configuration entry to disable/enable MWAIT/MONITOR instruction set as follows:

Enable Monitor Feature in BIOS?

With the above findings, it seemed clear the monitor instruction set in the customer’s new generation CPU was disabled out of the box. We coordinated with the customer to enable this “Monitor/MWAIT” feature in BIOS, however, they reported the BIOS had no configuration entry for this feature. As we could not enable it in BIOS setting, we continued to look for solutions.

Forcefully Enable Monitor Feature by IA32_MISC_ENABLE MSR?

The MWAIT/MONITOR feature can be enabled by forcefully setting IA32_MISC_ENABLE[bit 18] to 1 to make a new CPU feature list showing the monitor feature for a successful live migration. However, considering we didn’t truly understand the reason why this feature was disabled, there were concerns that doing this might lead to a potential unknown bug or risk system stability or performance. We realised a safer solution was needed.

Modify OS Kernel Option or Hack OpenStack?

The customer also asked if it was possible to work around this issue by changing OS kernel options or hacking OpenStack. Per our understanding, the solution is not in the layer of OS or OpenStack. We decided that we should look for other solutions...

Hack Libvirt?

Now we go up higher to libvirt layer. Per our investigation, libvirt supports 3 optional modes to configure a guest CPU.

  • custom
    This is the default mode with no specified mode attribute. In this mode, a persistent guest will see the same hardware, no matter what host the guest is using.
  • host-model
    The host-model mode is a shortcut to copy the host CPU definition from the capabilities XML (before v4.7.0 /usr/share/libvirt/cpu_map.xml) into domain XML. Since the CPU definition is copied before starting a domain, exactly the same XML can be used on different hosts while still providing the best guest CPU each host supports. Because it allows for a certain CPU difference between the destination node and the source node, it is the recommended mode for live migration. Specific flags may be enabled or disabled in addition to the host model, which may be used to fine tune features to be emulated.
  • host-passthrough
    In this mode, the CPU visible to the guest should be exactly the same as the host CPU, even in the aspects that libvirt does not understand. Migration of a guest using host-passthrough is dangerous if the source and destination hosts are not identical either in hardware or in configuration. If such a migration is attempted, the guest may hang or crash upon resuming execution on the destination host.

The customer is using host-model mode, so it is feasible to disable monitor in guest CPU configuration file by the following method.

<cpu mode='host-model/>
       <model fallback=forbid>SandyBridge</model>
   <feature policy='disable' name='monitor'/>
</cpu>

We also confirmed that the use of the “MWAIT/MONITOR” instruction requires checking availability on the target CPU, and when following this rule, it is safe even if disabling the monitor feature in the guest CPU.

We proposed a solution of hacking libvirt and bypassing the monitor feature check in order to make a successful live migration.

Issue Solution and Recommendation

Customer Experiment and Feedback

The customer implemented the suggested solutions of hacking libvirt and bypassing the monitor feature check, and they were able to successfully complete their live migration.

Recommendation for OpenStack Admin

Typically, host-model is the most popular guest CPU mode in the data center. We found that disabling the CPU monitor feature made live migration run smoothly. If the CPU monitor feature is not absolutely necessary, we recommend that it is disabled in host-model description.

Be aware that if the monitor feature was not disabled when starting a VM, then it will impact live migration to a host. The monitor feature of the “SandyBridge” model must be removed from the configuration file of cpu_map.xml (before v4.7.0) and the libvirt daemon must be restarted. The guest must also be restarted without the libvirt source code hack.

Note: The cpu_map.xml file was moved to /usr/share/libvirt/cpu_map/index.xml after version v4.7.0.

Recommendation for libvirt

Other customers that experience live migration failure due to this reason can submit an upgrade proposal to the libvirt community, suggesting a monitor feature compatibility check should return a warning if it fails, instead of an error that blocks live migration, because the monitor feature can be disabled or can disappear in a future platform. The premise is that the user fully understands the CPU monitor feature and understands the risk of skipping this check.

Summary

Intel is a customer-oriented company that has taken every effort to maintain compatibility among generations and product family, and provides many types of support to satisfy customer requirements. Occasionally, customers will encounter feature compatibility issues during application migration or VM migration. A commonly used method is to investigate from the low-level BIOS to high-level Library/Application, or vice-versa, to locate and understand the true cause of the issue.

After investigation, many issues can be simply solved by a) modifying BIOS options or b) changing OS kernel options. Very few of them require the amount of effort that was described in this article. We hope that this article can be used as a reference to solve the same or similar issues that may be encountered by customers.

Acknowledgements

Several Intel colleagues and a technical expert on the customer side have given valuable comments and ideas to this paper:
Xu, Hejie (Intel) is an OpenStack Nova core developer and provided source code walk-through assistance.
Wang, Jerry (Intel) offered BIOS consultation.
Wang, Huaqiang (Intel) contributed libvirt advice.

References