Vmware offline migration #2848

DaanHoogland · 2018-09-18T14:40:55Z

Description

Enable storage migration for stopped VMs on Vmware and for detached volumes on primary storages in Vmware clusters.

see https://cwiki.apache.org/confluence/display/CLOUDSTACK/VMware+offline+migrations

Types of changes

Breaking change (fix or feature that would cause existing functionality to change)
New feature (non-breaking change which adds functionality)
Bug fix (non-breaking change which fixes an issue)
Enhancement (improves an existing feature and functionality)
Cleanup (Code refactoring and cleanup, that may add test cases)

GitHub Issue/PRs

Screenshots (if appropriate):

How Has This Been Tested?

Checklist:

I have read the CONTRIBUTING document.
My code follows the code style of this project.
My change requires a change to the documentation.
I have updated the documentation accordingly.
Testing
I have added tests to cover my changes.
All relevant new and existing integration tests have passed.
A full integration testsuite with all test that can run on my environment has passed.

blueorangutan · 2018-09-19T21:07:00Z

@DaanHoogland a Trillian-Jenkins matrix job (centos6 mgmt + xs71, centos7 mgmt + vmware65, centos7 mgmt + kvmcentos7) has been kicked to run smoke tests

DaanHoogland · 2018-09-20T08:55:51Z

@blueorangutan package

blueorangutan · 2018-09-20T08:56:03Z

@DaanHoogland a Jenkins job has been kicked to build packages. I'll keep you posted as I make progress.

blueorangutan · 2018-09-20T09:31:23Z

Packaging result: ✔centos6 ✔centos7 ✔debian. JID-2312

DaanHoogland · 2018-09-20T09:32:29Z

travis red, but the travis job says all green ???
rekicking

blueorangutan · 2018-09-20T09:35:03Z

@DaanHoogland unsupported parameters provided. Supported mgmt server os are: centos6, centos7, ubuntu. Supported hypervisors are: kvm-centos6, kvm-centos7, kvm-ubuntu, xenserver-71, xenserver-65sp1, xenserver-62sp1, vmware-65, vmware-60u2, vmware-55u3, vmware-51u1, vmware-50u1

DaanHoogland · 2018-09-20T10:16:27Z

@blueorangutan test centos7 vmware-55u3

blueorangutan · 2018-09-20T10:17:03Z

@DaanHoogland a Trillian-Jenkins test job (centos7 mgmt + vmware-55u3) has been kicked to run smoke tests

rohityadavcloud · 2018-09-20T11:27:48Z

all environments have failed deployment, see sblab for errors @DaanHoogland

blueorangutan · 2018-09-20T19:06:54Z

Trillian test result (tid-3034)
Environment: vmware-55u3 (x2), Advanced Networking with Mgmt server 7
Total time taken: 30002 seconds
Marvin logs: https://github.com/blueorangutan/acs-prs/releases/download/trillian/pr2848-t3034-vmware-55u3.zip
Intermittent failure detected: /marvin/tests/smoke/test_deploy_virtio_scsi_vm.py
Intermittent failure detected: /marvin/tests/smoke/test_multipleips_per_nic.py
Intermittent failure detected: /marvin/tests/smoke/test_snapshots.py
Intermittent failure detected: /marvin/tests/smoke/test_vm_life_cycle.py
Intermittent failure detected: /marvin/tests/smoke/test_host_maintenance.py
Smoke tests completed. 64 look OK, 5 have error(s)
Only failed tests results shown below:

Test	Result	Time (s)	Test File
ContextSuite context=TestDeployVirtioSCSIVM>:teardown	`Error`	0.00	test_deploy_virtio_scsi_vm.py
test_nic_secondaryip_add_remove	`Error`	105.63	test_multipleips_per_nic.py
test_02_list_snapshots_with_removed_data_store	`Error`	54.41	test_snapshots.py
runTest	`Error`	0.00	test_vm_life_cycle.py
test_02_cancel_host_maintenace_with_migration_jobs	`Error`	1.24	test_host_maintenance.py

DaanHoogland · 2018-09-21T08:32:16Z

@blueorangutan test matrix

DaanHoogland · 2018-09-22T07:45:24Z

@blueorangutan test matrix

blueorangutan · 2018-09-22T07:46:12Z

@DaanHoogland a Trillian-Jenkins matrix job (centos6 mgmt + xs71, centos7 mgmt + vmware65, centos7 mgmt + kvmcentos7) has been kicked to run smoke tests

blueorangutan · 2018-09-22T14:07:41Z

Trillian test result (tid-3043)
Environment: kvm-centos7 (x2), Advanced Networking with Mgmt server 7
Total time taken: 21013 seconds
Marvin logs: https://github.com/blueorangutan/acs-prs/releases/download/trillian/pr2848-t3043-kvm-centos7.zip
Intermittent failure detected: /marvin/tests/smoke/test_multipleips_per_nic.py
Intermittent failure detected: /marvin/tests/smoke/test_privategw_acl.py
Intermittent failure detected: /marvin/tests/smoke/test_vm_life_cycle.py
Intermittent failure detected: /marvin/tests/smoke/test_hostha_kvm.py
Smoke tests completed. 65 look OK, 4 have error(s)
Only failed tests results shown below:

Test	Result	Time (s)	Test File
test_nic_secondaryip_add_remove	`Error`	26.33	test_multipleips_per_nic.py
test_03_vpc_privategw_restart_vpc_cleanup	`Failure`	1122.50	test_privategw_acl.py
runTest	`Error`	0.00	test_vm_life_cycle.py
test_hostha_enable_ha_when_host_in_maintenance	`Error`	2.42	test_hostha_kvm.py

blueorangutan · 2018-09-22T14:15:07Z

Trillian test result (tid-3042)
Environment: xenserver-71 (x2), Advanced Networking with Mgmt server 7
Total time taken: 21204 seconds
Marvin logs: https://github.com/blueorangutan/acs-prs/releases/download/trillian/pr2848-t3042-xenserver-71.zip
Intermittent failure detected: /marvin/tests/smoke/test_deploy_virtio_scsi_vm.py
Intermittent failure detected: /marvin/tests/smoke/test_multipleips_per_nic.py
Intermittent failure detected: /marvin/tests/smoke/test_scale_vm.py
Intermittent failure detected: /marvin/tests/smoke/test_vm_life_cycle.py
Smoke tests completed. 65 look OK, 4 have error(s)
Only failed tests results shown below:

Test	Result	Time (s)	Test File
ContextSuite context=TestDeployVirtioSCSIVM>:teardown	`Error`	0.00	test_deploy_virtio_scsi_vm.py
test_nic_secondaryip_add_remove	`Error`	42.99	test_multipleips_per_nic.py
test_01_scale_vm	`Error`	15.43	test_scale_vm.py
runTest	`Error`	0.00	test_vm_life_cycle.py

blueorangutan · 2018-09-22T15:07:38Z

Trillian test result (tid-3044)
Environment: vmware-65 (x2), Advanced Networking with Mgmt server 7
Total time taken: 24611 seconds
Marvin logs: https://github.com/blueorangutan/acs-prs/releases/download/trillian/pr2848-t3044-vmware-65.zip
Intermittent failure detected: /marvin/tests/smoke/test_deploy_virtio_scsi_vm.py
Intermittent failure detected: /marvin/tests/smoke/test_deploy_vm_root_resize.py
Intermittent failure detected: /marvin/tests/smoke/test_multipleips_per_nic.py
Intermittent failure detected: /marvin/tests/smoke/test_snapshots.py
Intermittent failure detected: /marvin/tests/smoke/test_vm_life_cycle.py
Smoke tests completed. 65 look OK, 4 have error(s)
Only failed tests results shown below:

Test	Result	Time (s)	Test File
ContextSuite context=TestDeployVirtioSCSIVM>:teardown	`Error`	0.00	test_deploy_virtio_scsi_vm.py
test_nic_secondaryip_add_remove	`Error`	63.57	test_multipleips_per_nic.py
test_02_list_snapshots_with_removed_data_store	`Error`	46.78	test_snapshots.py
runTest	`Error`	0.00	test_vm_life_cycle.py

dhlaluku · 2019-01-21T12:01:14Z

@rafaelweingartner do we have any outstanding items for this? /CC @GabrielBrascher

rafaelweingartner · 2019-01-21T12:09:11Z

@dhlaluku I am seeing this error in Travis:

raise Exception("Cleanup failed with %s" % e)
Exception: Cleanup failed with Execute cmd: deletestoragepool failed, due to: errorCode: 530, errorText:Cannot delete pool Primary XEN as there are associated non-destroyed vols for this pool
-------------------- >> begin captured logging << --------------------

Can you check it?

dhlaluku · 2019-01-23T07:21:32Z

`nosetests --with-xunit --xunit-file=results.xml --with-marvin --marvin-config=setup/dev/advanced.cfg -s -a tags=advanced,require_hardware="false" --zone=Sandbox-Simulator --hypervisor=simulator test/integration/smoke/test_primary_storage.py

==== Marvin Init Started ====

=== Marvin Parse Config Successful ===

=== Marvin Setting TestData Successful===

==== Log Folder Path: /tmp/MarvinLogs/Jan_23_2019_09_14_40_6K9K05. All logs will be available here ====

=== Marvin Init Logging Successful===

==== Marvin Init Successful ====
=== TestName: test_01_add_primary_storage_disabled_host | Status : SUCCESS ===

=== TestName: test_01_primary_storage_nfs | Status : SUCCESS ===

=== TestName: test_01_deploy_vms_storage_tags | Status : SUCCESS ===

=== TestName: test_02_edit_primary_storage_tags | Status : SUCCESS ===

=== TestName: test_03_migration_options_storage_tags | Status : SUCCESS ===

===final results are now copied to: /tmp/MarvinLogs/test_primary_storage_NFVHD7===
`
@rafaelweingartner not sure why Travis CI is failing but tests are passing from my local simulator environment

rafaelweingartner · 2019-01-23T10:34:20Z

The error changed now. did you rebase or something?

Traceback (most recent call last):
  File "/home/travis/.local/lib/python2.7/site-packages/nose/suite.py", line 209, in run
    self.setUp()
  File "/home/travis/.local/lib/python2.7/site-packages/nose/suite.py", line 292, in setUp
    self.setupContext(ancestor)
  File "/home/travis/.local/lib/python2.7/site-packages/nose/suite.py", line 315, in setupContext
    try_run(context, names)
  File "/home/travis/.local/lib/python2.7/site-packages/nose/util.py", line 471, in try_run
    return func()
  File "/home/travis/build/apache/cloudstack/test/integration/smoke/test_deploy_vm_root_resize.py", line 55, in setUpClass
    cls.services["mode"] = cls.zone.networktype
AttributeError: 'str' object has no attribute 'networktype'

dhlaluku · 2019-01-23T10:40:24Z

@rafaelweingartner I did a rebase against the latest master and force-pushed. The Travis CI errors are not related to this PR and they keep on changing every time I do a force-push. I have tested them locally against simulator based env with Ubuntu 18.04.

It is probably worth noting that they might be related to the environments with which they run, I notice that they run against Debian trusty ~ Ubuntu 14.04. We should probably upgrade to at least Ubuntu 16.04 LTS

- Also add VM disk consolidation call on successful VM migrations

* Refactor debug messages to use String.format() when debug log level is enabled.

…ly selecting the first one

…pool migration

…ports

…to favour the use of the ApacheUtils

dhlaluku · 2019-01-23T18:02:51Z

@blueorangutan package

blueorangutan · 2019-01-23T18:03:43Z

@dhlaluku a Jenkins job has been kicked to build packages. I'll keep you posted as I make progress.

blueorangutan · 2019-01-23T18:27:06Z

Packaging result: ✔centos6 ✔centos7 ✔debian. JID-2560

dhlaluku · 2019-01-23T18:54:49Z

@blueorangutan test centos7 vmware-65

blueorangutan · 2019-01-23T18:55:43Z

@dhlaluku a Trillian-Jenkins test job (centos7 mgmt + vmware-65) has been kicked to run smoke tests

blueorangutan · 2019-01-24T06:45:35Z

Trillian test result (tid-3340)
Environment: vmware-65 (x2), Advanced Networking with Mgmt server 7
Total time taken: 39883 seconds
Marvin logs: https://github.com/blueorangutan/acs-prs/releases/download/trillian/pr2848-t3340-vmware-65.zip
Intermittent failure detected: /marvin/tests/smoke/test_primary_storage.py
Intermittent failure detected: /marvin/tests/smoke/test_privategw_acl.py
Intermittent failure detected: /marvin/tests/smoke/test_snapshots.py
Intermittent failure detected: /marvin/tests/smoke/test_vm_life_cycle.py
Intermittent failure detected: /marvin/tests/smoke/test_vpc_redundant.py
Intermittent failure detected: /marvin/tests/smoke/test_host_maintenance.py
Smoke tests completed. 64 look OK, 6 have error(s)
Only failed tests results shown below:

Test	Result	Time (s)	Test File
test_01_add_primary_storage_disabled_host	`Error`	0.51	test_primary_storage.py
test_01_primary_storage_nfs	`Error`	0.13	test_primary_storage.py
ContextSuite context=TestStorageTags>:setup	`Error`	0.25	test_primary_storage.py
test_03_vpc_privategw_restart_vpc_cleanup	`Error`	655.04	test_privategw_acl.py
test_02_list_snapshots_with_removed_data_store	`Error`	1.17	test_snapshots.py
test_11_migrate_vm	`Error`	138.73	test_vm_life_cycle.py
test_05_rvpc_multi_tiers	`Failure`	501.70	test_vpc_redundant.py
test_05_rvpc_multi_tiers	`Error`	544.30	test_vpc_redundant.py
test_01_cancel_host_maintenace_with_no_migration_jobs	`Failure`	4.32	test_host_maintenance.py
test_02_cancel_host_maintenace_with_migration_jobs	`Failure`	227.69	test_host_maintenance.py

dhlaluku · 2019-01-24T07:18:33Z

@blueorangutan test centos7 vmware-65

blueorangutan · 2019-01-24T07:18:45Z

@dhlaluku a Trillian-Jenkins test job (centos7 mgmt + vmware-65) has been kicked to run smoke tests

blueorangutan · 2019-01-24T17:13:32Z

Trillian test result (tid-3342)
Environment: vmware-65 (x2), Advanced Networking with Mgmt server 7
Total time taken: 34194 seconds
Marvin logs: https://github.com/blueorangutan/acs-prs/releases/download/trillian/pr2848-t3342-vmware-65.zip
Smoke tests completed. 70 look OK, 0 have error(s)
Only failed tests results shown below:

Test	Result	Time (s)	Test File

GabrielBrascher

LGTM

* - Offline VM and Volume migration on Vmware hypervisor hosts - Also add VM disk consolidation call on successful VM migrations * Fix indentation of marvin test file and reformat against PEP8 * * Fix few comment typos * Refactor debug messages to use String.format() when debug log level is enabled. * Send list of commands returned by hypervisor Guru instead of explicitly selecting the first one * Fix unhandled NPE during VM migration * Revert back to distinct event descriptions for VM to host or storage pool migration * Reformat test_primary_storage file against PEP-8 and Remove unused imports * Revert back the deprecation messages in the custom StringUtils class to favour the use of the ApacheUtils

…louds' Substituição do logotipo kubernetes na interface gráfica Closes apache#2848 See merge request scclouds/scclouds!1237

DaanHoogland force-pushed the VmwareOfflineMigration branch from 6e6a092 to 75ec902 Compare September 18, 2018 14:58

rafaelweingartner assigned DaanHoogland Sep 18, 2018

rafaelweingartner added type:enhancement component:vmware labels Sep 18, 2018

rafaelweingartner added this to the 4.12.0.0 milestone Sep 18, 2018

DaanHoogland force-pushed the VmwareOfflineMigration branch 2 times, most recently from 03ee23d to 2db4bda Compare September 19, 2018 12:30

apache deleted a comment from blueorangutan Sep 19, 2018

apache deleted a comment from blueorangutan Sep 20, 2018

DaanHoogland requested review from mike-tutkowski and rafaelweingartner September 21, 2018 07:06

DaanHoogland changed the title ~~WIP: Vmware offline migration~~ Vmware offline migration Sep 22, 2018

dhlaluku force-pushed the VmwareOfflineMigration branch from f9ae439 to a8599e8 Compare January 21, 2019 12:15

dhlaluku force-pushed the VmwareOfflineMigration branch from 84ff31e to fc25c2d Compare January 23, 2019 08:09

borisstoyanov and others added 8 commits January 23, 2019 17:56

- Offline VM and Volume migration on Vmware hypervisor hosts

17a86ef

- Also add VM disk consolidation call on successful VM migrations

Fix indentation of marvin test file and reformat against PEP8

103a0b2

* Fix few comment typos

1c552b3

* Refactor debug messages to use String.format() when debug log level is enabled.

Send list of commands returned by hypervisor Guru instead of explicit…

e3956e2

…ly selecting the first one

Fix unhandled NPE during VM migration

2e5e58c

Revert back to distinct event descriptions for VM to host or storage …

52abf56

…pool migration

Reformat test_primary_storage file against PEP-8 and Remove unused im…

4673035

…ports

Revert back the deprecation messages in the custom StringUtils class …

918b02f

…to favour the use of the ApacheUtils

dhlaluku force-pushed the VmwareOfflineMigration branch from fc25c2d to 918b02f Compare January 23, 2019 16:40

GabrielBrascher approved these changes Jan 24, 2019

View reviewed changes

GabrielBrascher merged commit b363fd4 into apache:master Jan 25, 2019

RodrigoDLopez mentioned this pull request Aug 24, 2020

Removes unnecessary validations #4283

Closed

5 tasks

Vmware offline migration #2848

Vmware offline migration #2848

Uh oh!

Conversation

DaanHoogland commented Sep 18, 2018 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Description

Types of changes

GitHub Issue/PRs

Screenshots (if appropriate):

How Has This Been Tested?

Checklist:

Uh oh!

blueorangutan commented Sep 19, 2018

Uh oh!

DaanHoogland commented Sep 20, 2018

Uh oh!

blueorangutan commented Sep 20, 2018

Uh oh!

blueorangutan commented Sep 20, 2018

Uh oh!

DaanHoogland commented Sep 20, 2018 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

blueorangutan commented Sep 20, 2018

Uh oh!

DaanHoogland commented Sep 20, 2018

Uh oh!

blueorangutan commented Sep 20, 2018

Uh oh!

rohityadavcloud commented Sep 20, 2018

Uh oh!

blueorangutan commented Sep 20, 2018

Uh oh!

DaanHoogland commented Sep 21, 2018

Uh oh!

DaanHoogland commented Sep 22, 2018

Uh oh!

blueorangutan commented Sep 22, 2018

Uh oh!

blueorangutan commented Sep 22, 2018

Uh oh!

blueorangutan commented Sep 22, 2018

Uh oh!

blueorangutan commented Sep 22, 2018

Uh oh!

dhlaluku commented Jan 21, 2019

Uh oh!

rafaelweingartner commented Jan 21, 2019

Uh oh!

dhlaluku commented Jan 23, 2019

Uh oh!

rafaelweingartner commented Jan 23, 2019

Uh oh!

dhlaluku commented Jan 23, 2019

Uh oh!

dhlaluku commented Jan 23, 2019

Uh oh!

blueorangutan commented Jan 23, 2019

Uh oh!

blueorangutan commented Jan 23, 2019

Uh oh!

dhlaluku commented Jan 23, 2019

Uh oh!

blueorangutan commented Jan 23, 2019

Uh oh!

blueorangutan commented Jan 24, 2019

Uh oh!

dhlaluku commented Jan 24, 2019

Uh oh!

blueorangutan commented Jan 24, 2019

Uh oh!

blueorangutan commented Jan 24, 2019

Uh oh!

GabrielBrascher left a comment

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

DaanHoogland commented Sep 18, 2018 •

edited

Loading

DaanHoogland commented Sep 20, 2018 •

edited

Loading