[KVM] Fix VM migration error due to VNC password on libvirt limiting versions #6404

nvazquez · 2022-05-22T03:12:36Z

Description

This PR fixes a VM migration issue described on #6402 (comment)

Types of changes

Breaking change (fix or feature that would cause existing functionality to change)
New feature (non-breaking change which adds functionality)
Bug fix (non-breaking change which fixes an issue)
Enhancement (improves an existing feature and functionality)
Cleanup (Code refactoring and cleanup, that may add test cases)

Feature/Enhancement Scale or Bug Severity

Feature/Enhancement Scale

Major
Minor

Bug Severity

Screenshots (if appropriate):

How Has This Been Tested?

Initial state: CS 4.16.1 / 2 KVM hosts running OpenSUSE 15.3 - libvirt 7.1.0 and QEMU 5.2.0
Create 3 VMs
Put host 2 on maintenance
Upgrade host packages from: https://software.opensuse.org/download/package?package=libvirt&project=Virtualization to libvirt 8.3.0 and QEMU 6.2.0
Cancel maintenance on host 2
Attempt migrating any VM to host 2 -> FAILURE:

2022-05-22 03:56:07,297 DEBUG [c.c.c.CapacityManagerImpl] (Work-Job-Executor-8:ctx-77987975 job-41/job-42 ctx-91220c68) (logid:c4ad92ef) VM instance {id: "5", name: "i-2-5-VM", uuid: 
"6a2339af-5236-49aa-b9a0-65a2c9ce3952", type="User"} state transited from [Migrating] to [Running] with event [OperationFailed]. VM's original host: Host {"id": "1", "name": "ref-trl-
3127-k-M7-nicolas-vazquez-kvm1", "uuid": "9c851ff3-06f5-46e0-a822-f029a375748a", "type"="Routing"}, new host: Host {"id": "1", "name": "ref-trl-3127-k-M7-nicolas-vazquez-kvm1", "uuid"
: "9c851ff3-06f5-46e0-a822-f029a375748a", "type"="Routing"}, host before state transition: Host {"id": "4", "name": "ref-trl-3141-k-M7-nicolas-vazquez-kvm1", "uuid": "90107936-3a2d-48
06-b5af-561e97620664", "type"="Routing"}
2022-05-22 03:56:07,300 DEBUG [c.c.c.CapacityManagerImpl] (Work-Job-Executor-8:ctx-77987975 job-41/job-42 ctx-91220c68) (logid:c4ad92ef) Hosts's actual total CPU: 6300 and CPU after a
pplying overprovisioning: 12600
2022-05-22 03:56:07,301 DEBUG [c.c.c.CapacityManagerImpl] (Work-Job-Executor-8:ctx-77987975 job-41/job-42 ctx-91220c68) (logid:c4ad92ef) Hosts's actual total RAM: (6.77 GB) 7264485376
 and RAM after applying overprovisioning: (6.77 GB) 7264485376
2022-05-22 03:56:07,301 DEBUG [c.c.c.CapacityManagerImpl] (Work-Job-Executor-8:ctx-77987975 job-41/job-42 ctx-91220c68) (logid:c4ad92ef) release cpu from host: 4, old used: 500,reserv
ed: 0, actual total: 6300, total with overprovisioning: 12600; new used: 0,reserved:0; movedfromreserved: false,moveToReserveredfalse
2022-05-22 03:56:07,301 DEBUG [c.c.c.CapacityManagerImpl] (Work-Job-Executor-8:ctx-77987975 job-41/job-42 ctx-91220c68) (logid:c4ad92ef) release mem from host: 4, old used: (512.00 MB
) 536870912,reserved: (0 bytes) 0, total: (6.77 GB) 7264485376; new used: (0 bytes) 0,reserved:(0 bytes) 0; movedfromreserved: false,moveToReserveredfalse
2022-05-22 03:56:07,305 ERROR [c.c.v.VmWorkJobHandlerProxy] (Work-Job-Executor-8:ctx-77987975 job-41/job-42 ctx-91220c68) (logid:c4ad92ef) Invocation exception, caused by: com.cloud.u
tils.exception.CloudRuntimeException: Exception during migrate: org.libvirt.LibvirtException: unsupported configuration: VNC password is 22 characters long, only 8 permitted
2022-05-22 03:56:07,305 INFO  [c.c.v.VmWorkJobHandlerProxy] (Work-Job-Executor-8:ctx-77987975 job-41/job-42 ctx-91220c68) (logid:c4ad92ef) Rethrow exception com.cloud.utils.exception.
CloudRuntimeException: Exception during migrate: org.libvirt.LibvirtException: unsupported configuration: VNC password is 22 characters long, only 8 permitted
2022-05-22 03:56:07,309 DEBUG [c.c.v.VmWorkJobDispatcher] (Work-Job-Executor-8:ctx-77987975 job-41/job-42) (logid:c4ad92ef) Done with run of VM work job: com.cloud.vm.VmWorkMigrate fo
r VM 5, job origin: 41
2022-05-22 03:56:07,310 ERROR [c.c.v.VmWorkJobDispatcher] (Work-Job-Executor-8:ctx-77987975 job-41/job-42) (logid:c4ad92ef) Unable to complete AsyncJobVO: {id:42, userId: 2, accountId
: 2, instanceType: null, instanceId: null, cmd: com.cloud.vm.VmWorkMigrate, cmdInfo: rO0ABXNyABpjb20uY2xvdWQudm0uVm1Xb3JrTWlncmF0ZRdxQXtPtzYqAgAGSgAJc3JjSG9zdElkTAAJY2x1c3RlcklkdAAQTG
phdmEvbGFuZy9Mb25nO0wABmhvc3RJZHEAfgABTAAFcG9kSWRxAH4AAUwAB3N0b3JhZ2V0AA9MamF2YS91dGlsL01hcDtMAAZ6b25lSWRxAH4AAXhyABNjb20uY2xvdWQudm0uVm1Xb3Jrn5m2VvAlZ2sCAARKAAlhY2NvdW50SWRKAAZ1c2VyS
WRKAAR2bUlkTAALaGFuZGxlck5hbWV0ABJMamF2YS9sYW5nL1N0cmluZzt4cAAAAAAAAAACAAAAAAAAAAIAAAAAAAAABXQAGVZpcnR1YWxNYWNoaW5lTWFuYWdlckltcGwAAAAAAAAAAXNyAA5qYXZhLmxhbmcuTG9uZzuL5JDMjyPfAgABSgAF
dmFsdWV4cgAQamF2YS5sYW5nLk51bWJlcoaslR0LlOCLAgAAeHAAAAAAAAAAAXNxAH4ABwAAAAAAAAAEcQB-AAlwcQB-AAk, cmdVersion: 0, status: IN_PROGRESS, processStatus: 0, resultCode: 0, result: null, ini
tMsid: 32986154140800, completeMsid: null, lastUpdated: null, lastPolled: null, created: Sun May 22 03:56:06 UTC 2022, removed: null}, job origin:41
com.cloud.utils.exception.CloudRuntimeException: Exception during migrate: org.libvirt.LibvirtException: unsupported configuration: VNC password is 22 characters long, only 8 permitte
d
        at com.cloud.vm.VirtualMachineManagerImpl.migrate(VirtualMachineManagerImpl.java:2795)
        at com.cloud.vm.VirtualMachineManagerImpl.orchestrateMigrate(VirtualMachineManagerImpl.java:2658)
        at com.cloud.vm.VirtualMachineManagerImpl.orchestrateMigrate(VirtualMachineManagerImpl.java:5883)
        at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.base/java.lang.reflect.Method.invoke(Method.java:566)

Upgrade cloudstack-agent packages on host 1, containing this fix -> then upgrade host 1
Attempt migrations to host 2 -> SUCCESS

…versions

nvazquez · 2022-05-22T03:12:56Z

@blueorangutan package

blueorangutan · 2022-05-22T03:16:03Z

@nvazquez a Jenkins job has been kicked to build packages. It will be bundled with KVM, XenServer and VMware SystemVM templates. I'll keep you posted as I make progress.

blueorangutan · 2022-05-22T04:08:11Z

Packaging result: ✔️ el7 ✔️ el8 ✔️ debian ✔️ suse15. SL-JID 3466

nvazquez · 2022-05-22T04:38:21Z

@blueorangutan package

blueorangutan · 2022-05-22T04:39:02Z

@nvazquez a Jenkins job has been kicked to build packages. It will be bundled with KVM, XenServer and VMware SystemVM templates. I'll keep you posted as I make progress.

blueorangutan · 2022-05-22T05:15:44Z

Packaging result: ✔️ el7 ✔️ el8 ✔️ debian ✔️ suse15. SL-JID 3468

nvazquez · 2022-05-22T11:14:27Z

@blueorangutan test

blueorangutan · 2022-05-22T11:15:03Z

@nvazquez a Trillian-Jenkins test job (centos7 mgmt + kvm-centos7) has been kicked to run smoke tests

leolleeooleo · 2022-05-22T11:25:34Z

It works.
But, not host 2, shoud be host 1.

Upgrade cloudstack-agent packages on host 2, containing this fix -> then upgrade host 1

weizhouapache · 2022-05-22T13:35:29Z

It works. But, not host 2, shoud be host 1.

Upgrade cloudstack-agent packages on host 2, containing this fix -> then upgrade host 1

@leolleeooleo
The source host should have this fix before vm migration.

leolleeooleo · 2022-05-22T14:28:25Z

It works. But, not host 2, shoud be host 1.

Upgrade cloudstack-agent packages on host 2, containing this fix -> then upgrade host 1

@leolleeooleo The source host should have this fix before vm migration.

@weizhouapache
That's what i mean.
In @nvazquez test, the source host is host 1.

nvazquez · 2022-05-22T15:04:45Z

Yes, was a mistake on the PR description, have updated it - thanks for testing @leolleeooleo. Please submit your review under Files changed -> Review

weizhouapache

code lgtm

...vm/src/main/java/com/cloud/hypervisor/kvm/resource/wrapper/LibvirtMigrateCommandWrapper.java

leolleeooleo · 2022-05-22T16:26:55Z

Yes, was a mistake on the PR description, have updated it - thanks for testing @leolleeooleo. Please submit your review under Files changed -> Review

"then upgrade host 1" is right.

Upgrade cloudstack-agent packages on host 1, containing this fix -> then upgrade host 1

...vm/src/main/java/com/cloud/hypervisor/kvm/resource/wrapper/LibvirtMigrateCommandWrapper.java

blueorangutan · 2022-05-22T19:45:38Z

Trillian test result (tid-4228)
Environment: kvm-centos7 (x2), Advanced Networking with Mgmt server 7
Total time taken: 29318 seconds
Marvin logs: https://github.com/blueorangutan/acs-prs/releases/download/trillian/pr6404-t4228-kvm-centos7.zip
Smoke tests completed. 92 look OK, 0 have errors
Only failed tests results shown below:

Test	Result	Time (s)	Test File

nvazquez · 2022-05-23T02:59:10Z

Thanks for the review @leolleeooleo @weizhouapache - I have refactored it to a simpler approach

nvazquez · 2022-05-23T03:02:13Z

@blueorangutan package

nvazquez · 2022-05-23T03:49:57Z

@blueorangutan package

blueorangutan · 2022-05-23T03:51:03Z

@nvazquez a Jenkins job has been kicked to build packages. It will be bundled with KVM, XenServer and VMware SystemVM templates. I'll keep you posted as I make progress.

blueorangutan · 2022-05-23T04:32:46Z

Packaging result: ✔️ el7 ✔️ el8 ✔️ debian ✔️ suse15. SL-JID 3471

weizhouapache · 2022-05-23T06:16:30Z

@blueorangutan package

blueorangutan · 2022-05-23T06:18:03Z

@weizhouapache a Jenkins job has been kicked to build packages. It will be bundled with KVM, XenServer and VMware SystemVM templates. I'll keep you posted as I make progress.

weizhouapache

code lgtm

blueorangutan · 2022-05-23T06:54:57Z

Packaging result: ✔️ el7 ✔️ el8 ✔️ debian ✔️ suse15. SL-JID 3472

weizhouapache · 2022-05-23T07:01:32Z

@blueorangutan test

blueorangutan · 2022-05-23T07:02:03Z

@weizhouapache a Trillian-Jenkins test job (centos7 mgmt + kvm-centos7) has been kicked to run smoke tests

weizhouapache

manually tested OK

DaanHoogland

clgtm

DaanHoogland · 2022-05-23T08:13:02Z

...vm/src/main/java/com/cloud/hypervisor/kvm/resource/wrapper/LibvirtMigrateCommandWrapper.java

+            // Limit the VNC password in case the length is greater than 8 characters
+            // Since libvirt version 8 VNC passwords are limited to 8 characters
+            String vncPassword = org.apache.commons.lang3.StringUtils.truncate(to.getVncPassword(), 8);
+            xmlDesc = replaceIpForVNCInDescFileAndNormalizePassword(xmlDesc, target, vncPassword);


separate methos?

Suggested change

// Limit the VNC password in case the length is greater than 8 characters

// Since libvirt version 8 VNC passwords are limited to 8 characters

String vncPassword = org.apache.commons.lang3.StringUtils.truncate(to.getVncPassword(), 8);

xmlDesc = replaceIpForVNCInDescFileAndNormalizePassword(xmlDesc, target, vncPassword);

/**

* Limit the VNC password in case the length is greater than 8 characters

* Since libvirt version 8 VNC passwords are limited to 8 characters

*/

private XmlDescOrSuchAThing getXmlDesc(....) {

String vncPassword = org.apache.commons.lang3.StringUtils.truncate(to.getVncPassword(), 8);

return replaceIpForVNCInDescFileAndNormalizePassword(xmlDesc, target, vncPassword);

}

blueorangutan · 2022-05-23T15:42:28Z

Trillian test result (tid-4229)
Environment: kvm-centos7 (x2), Advanced Networking with Mgmt server 7
Total time taken: 29967 seconds
Marvin logs: https://github.com/blueorangutan/acs-prs/releases/download/trillian/pr6404-t4229-kvm-centos7.zip
Smoke tests completed. 92 look OK, 0 have errors
Only failed tests results shown below:

Test	Result	Time (s)	Test File

…versions (apache#6404) * [KVM] Fix VM migration error due to VNC password on libvirt limiting versions * Fix passwd value * Simplify implementation

* Extract the IO_URING configuration into the agent.properties (apache#6253) When using advanced virtualization the IO Driver is not supported. The admin will decide if want to enable/disable this configuration from agent.properties file. The default value is true * kvm: truncate vnc password to 8 chars (apache#6244) This PR truncates the vnc password of kvm vms to 8 chars to support latest versions of libvirt. * merge fix Signed-off-by: Abhishek Kumar <[email protected]> * [KVM] Enable IOURING only when it is available on the host (apache#6399) * [KVM] Disable IOURING by default on agents * Refactor * Remove agent property for iouring * Restore property * Refactor suse check and enable on ubuntu by default * Refactor irrespective of guest OS * Improvement * Logs and new path * Refactor condition to enable iouring * Improve condition * Refactor property check * Improvement * Doc comment * Extend comment * Move method * Add log * [KVM] Fix VM migration error due to VNC password on libvirt limiting versions (apache#6404) * [KVM] Fix VM migration error due to VNC password on libvirt limiting versions * Fix passwd value * Simplify implementation Co-authored-by: slavkap <[email protected]> Co-authored-by: Wei Zhou <[email protected]> Co-authored-by: Nicolas Vazquez <[email protected]>

…versions (apache#6404) * [KVM] Fix VM migration error due to VNC password on libvirt limiting versions * Fix passwd value * Simplify implementation (cherry picked from commit b1c8b5a) Signed-off-by: Rohit Yadav <[email protected]>

[KVM] Fix VM migration error due to VNC password on libvirt limiting …

652bf25

…versions

nvazquez added type:bug component:migration labels May 22, 2022

nvazquez added this to the 4.17.0.0 milestone May 22, 2022

nvazquez requested review from Pearl1594, rohityadavcloud, shwstppr, slavkap and weizhouapache May 22, 2022 03:12

boring-cyborg bot added the component:kvm label May 22, 2022

nvazquez mentioned this pull request May 22, 2022

Backport: kvm: truncate vnc password to 8 chars (#6244) #6402

Merged

12 tasks

Fix passwd value

e9e07f9

weizhouapache approved these changes May 22, 2022

View reviewed changes

weizhouapache reviewed May 22, 2022

View reviewed changes

...vm/src/main/java/com/cloud/hypervisor/kvm/resource/wrapper/LibvirtMigrateCommandWrapper.java Outdated Show resolved Hide resolved

leolleeooleo reviewed May 22, 2022

View reviewed changes

...vm/src/main/java/com/cloud/hypervisor/kvm/resource/wrapper/LibvirtMigrateCommandWrapper.java Outdated Show resolved Hide resolved

...vm/src/main/java/com/cloud/hypervisor/kvm/resource/wrapper/LibvirtMigrateCommandWrapper.java Outdated Show resolved Hide resolved

Simplify implementation

85903ae

weizhouapache closed this May 23, 2022

weizhouapache reopened this May 23, 2022

weizhouapache approved these changes May 23, 2022

View reviewed changes

DaanHoogland approved these changes May 23, 2022

View reviewed changes

nvazquez merged commit b1c8b5a into apache:4.16 May 23, 2022

shwstppr mentioned this pull request Aug 12, 2022

Can not deploy VM using libvirt-8.0.0 ？ #6635

Closed

weizhouapache mentioned this pull request Jan 6, 2023

Secondary Storage System VM keeps crashing #7062

Closed

[KVM] Fix VM migration error due to VNC password on libvirt limiting versions #6404

[KVM] Fix VM migration error due to VNC password on libvirt limiting versions #6404

Uh oh!

Conversation

nvazquez commented May 22, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Description

Types of changes

Feature/Enhancement Scale or Bug Severity

Feature/Enhancement Scale

Bug Severity

Screenshots (if appropriate):

How Has This Been Tested?

Uh oh!

nvazquez commented May 22, 2022

Uh oh!

blueorangutan commented May 22, 2022

Uh oh!

blueorangutan commented May 22, 2022

Uh oh!

nvazquez commented May 22, 2022

Uh oh!

blueorangutan commented May 22, 2022

Uh oh!

blueorangutan commented May 22, 2022

Uh oh!

nvazquez commented May 22, 2022

Uh oh!

blueorangutan commented May 22, 2022

Uh oh!

leolleeooleo commented May 22, 2022

Uh oh!

weizhouapache commented May 22, 2022

Uh oh!

leolleeooleo commented May 22, 2022

Uh oh!

nvazquez commented May 22, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

weizhouapache left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

leolleeooleo commented May 22, 2022

Uh oh!

Uh oh!

Uh oh!

blueorangutan commented May 22, 2022

Uh oh!

nvazquez commented May 23, 2022

Uh oh!

nvazquez commented May 23, 2022

Uh oh!

nvazquez commented May 23, 2022

Uh oh!

blueorangutan commented May 23, 2022

Uh oh!

blueorangutan commented May 23, 2022

Uh oh!

weizhouapache commented May 23, 2022

Uh oh!

blueorangutan commented May 23, 2022

Uh oh!

weizhouapache left a comment

Choose a reason for hiding this comment

Uh oh!

blueorangutan commented May 23, 2022

Uh oh!

weizhouapache commented May 23, 2022

Uh oh!

blueorangutan commented May 23, 2022

Uh oh!

weizhouapache left a comment

Choose a reason for hiding this comment

Uh oh!

DaanHoogland left a comment

Choose a reason for hiding this comment

Uh oh!

DaanHoogland May 23, 2022

Choose a reason for hiding this comment

nvazquez commented May 22, 2022 •

edited

Loading

nvazquez commented May 22, 2022 •

edited

Loading