KVM NFS disk IO driver supporting IO_URING #5012
Conversation
|
@GabrielBrascher interesting PR, I checked the libvirt version. Which distros/version is this targeted against? |
|
@rhtyd as far as I know these Qemu and Libvirt packages can be found in Ubuntu 21.04. |
|
Thanks for explaining @GabrielBrascher good we're looking ahead. |
Especially with NVMe local storage in hypervisors this can vastly improve the overall I/O performance of Virtual Machines. See this comment from Gabriel with a graph of the performance boost possible: #4883 (comment) |
...ypervisors/kvm/src/main/java/com/cloud/hypervisor/kvm/resource/LibvirtComputingResource.java
Outdated
Show resolved
Hide resolved
plugins/hypervisors/kvm/src/main/java/com/cloud/hypervisor/kvm/resource/LibvirtVMDef.java
Outdated
Show resolved
Hide resolved
|
Thanks for the review, @GutoVeronezi. Code has been updated. |
|
I will check if I can install the required Libvirt/Qemu packages on Ubuntu 20.04 in order to test this implementation. I am not sure if they will be installable in 20.04 as we can find them on 21.04. If it works I can run some tests to check if all looks good when the XML is parsed and VM deployed. I will be able to run some tests on |
|
@blueorangutan package |
|
@rhtyd a Jenkins job has been kicked to build packages. I'll keep you posted as I make progress. |
|
Packaging result: ✖️ centos7 ✖️ centos8 ✔️ debian. SL-JID 271 |
|
@blueorangutan package |
|
@sureshanaparti a Jenkins job has been kicked to build packages. I'll keep you posted as I make progress. |
|
Packaging result: ✔️ centos7 ✔️ centos8 ✔️ debian. SL-JID 357 |
|
@blueorangutan test |
|
@sureshanaparti a Trillian-Jenkins test job (centos7 mgmt + kvm-centos7) has been kicked to run smoke tests |
|
Trillian test result (tid-1126)
|
|
I see unit tests @GabrielBrascher 👍 are integration tests for io driver-types in the planning/possible? or do we need hardware? (i don't think so, only kernel features and maybe qemu/libvirt version supporting it) |
| * (ii) Libvirt >= 6.3.0 | ||
| */ | ||
| protected void setDiskIoDriver(DiskDef disk) { | ||
| if (getHypervisorLibvirtVersion() >= HYPERVISOR_LIBVIRT_VERSION_SUPPORTS_IO_URING && getHypervisorQemuVersion() >= HYPERVISOR_QEMU_VERSION_SUPPORTS_IO_URING) { |
There was a problem hiding this comment.
LGTM and tests have passed, but this enables io_uring as default for the matching version of qemu and libvirt, could that cause an issue @GabrielBrascher for different type of guest OS?
There was a problem hiding this comment.
@rhtyd as far as I know there are no issues with different types of guest OS.
This would impact the Hypervisor side; thus, it is important that the virtualization layer supports it.
As far as I know, both Ubuntu and Centos would support as long as the Qemu >= 5.0 and Libvirt >= 6.3.0.
One issue that I have seen with this proposed implementation is when (live) migrating from a KVM node that supports IO_URING to one that does not. I will check how to update the IO Driver when migrating.
|
@DaanHoogland @rhtyd I've executed some manual tests on hardware that has the required Qemu & Libvirt versions to support IO_URING. The VM has the IO_URING added as expected in its XML when the versions match the requirements. @DaanHoogland Do you mean integration tests that would validate the domain XML according to the host qemu/libvirt version?. |
@GabrielBrascher it looks the performance are quite close, right ? |
|
I agree, @weizhouapache. I think that this is due to the tests being too simple, probably not the best way of exploring the IO optimizations with some "dd" tests. I think that I need to run some elaborated tests. There are some interesting tests thet show a great improvement in performance. For example: #4883 (comment) |
@GabrielBrascher |
yes, and does a copy/mv/del to verify io is functioning, maybe. I do not say we need it, just wondering. might be excessive. |
rohityadavcloud
left a comment
rohityadavcloud
left a comment
There was a problem hiding this comment.
Haven't tested, but no regressions so let's merge.
|
this PR prevent me from using Cloudstack |
|
same thing for router: |
|
@xlmnxp, can you share the libvirt/qemu versions and the OS? Also, do you use advanced virtualization? I had the same problem on Rocky with the advanced virtualization, and there is a fix which will be included in CS versions - 4.16.2.0 and 4.17. |
|
I indeed think this is due to the OS. Without knowing the OS and Qemu version we can't tell the root cause. |
OS:
I already disabled it in agent config but the router didn't start because of is_uring which I disabled
|
@xlmnxp |
Then how I can upgrade to 4.17? |
Is there any quick fix for it or patch because I want to try cloudstack in next 2 days? |
|
@xlmnxp, I could suggest until there isn't a released version with the fix, you can downgrade the qemu/libvirt versions to libvirt 6.0.0 and qemu 4.2 or something lower than the supported ones: |
Is there a way other then downgrade |
|
@xlmnxp, I don't have anything else on my mind :/ ... The most painless solution for me is downgrading the versions of libvirt/qemu. Another option is to switch to another OS (which supports io_uring) on the hypervisor or downgrade the CS version. |
then best option is to wait 4.17 |
|
I have same problem after I update to Rocky Linux 8.6 @xlmnxp , You can rebase #6253, #6399, and #6402 to tag 4.16.1.0 then replace jar file cloud-plugin-hypervisor-kvm-4.16.0.0.jar and cloud-agent-4.16.1.0.jar if you urgently need. My steps: clone the repository checkout #6399 and rebase to 4.16.1.0 and also need to fix VNC password as #6402 build jar files and copy to your agent You need to update CloudStack to 4.16.1.0 first You can use my jar file if you don't mind |
Great, That fix the issue 👍🏼 |
|
Thank you, I'm facing other issues, I will try to fix it Why there no discourse community forum or discord server that will help on troubleshooting and learn Cloudstack |
|
finally everything fixed and Cloudstack work great 👍🏼 |
12 participants
Description
Currently there is no disk IO driver configuration for VMs running on KVM. That's OK for most the cases; however, recently there have been added some quite interesting optimizations with the IO driver
io_uring.Note that IO URING requires:
Qemu >= 5.0, andLibvirt >= 6.3.0.By using
io_uringwe can see a massive I/O performance improvement within Virtual Machines running from Local and/or NFS storage.This implementation enhances the KVM disk configuration by adding workflow for setting the disk IO drivers. Additionally, if the Qemu and Libvirt versions matches with the required for having
io_uringwe are going to set it on the VM. If there is no support for such driver we keep it as it is nowadays, without any IO driver configured.Fixes: #4883
Types of changes
How Has This Been Tested?
Locally debugging with breakpoints on remote debug and asserting the IO driver as well as qemu/libvirt versions.