server: select root disk based on user input during vm import #4591

shwstppr · 2021-01-18T11:00:57Z

Description

Improves logic for selecting ROOT disk for VM to be imported when VM has multiple disks.
Based on user input of parameter - datadiskofferinglist, a check will be performed if user correctly provided disk offering mapping for n - 1 disks (one less than the total number of the disks). Remaining one disk will be considered as the ROOT disk.
This will allow correct ROOT disk being selected when first disk in the list of disks of listUnmanagedInstance API or list returned by vSphere SDK is not the ROOT disk.

Fixes #4547

Types of changes

Breaking change (fix or feature that would cause existing functionality to change)
New feature (non-breaking change which adds functionality)
Bug fix (non-breaking change which fixes an issue)
Enhancement (improves an existing feature and functionality)
Cleanup (Code refactoring and cleanup, that may add test cases)

Feature/Enhancement Scale or Bug Severity

Feature/Enhancement Scale

Major
Minor

Bug Severity

Screenshots (if appropriate):

How Has This Been Tested?

CMK

Unmanaged VM

> list unmanagedinstances clusterid=c4ae9767-cca3-46e3-a801-3218ea8b69d1
{
  "count": 1,
  "unmanagedinstance": [
    {
      "clusterid": "c4ae9767-cca3-46e3-a801-3218ea8b69d1",
      "cpucorepersocket": 1,
      "cpunumber": 1,
      "cpuspeed": 0,
      "disk": [
        {
          "capacity": 2147483648,
          "controller": "lsilogic",
          "controllerunit": 0,
          "datastorehost": "10.10.0.16",
          "datastorename": "3ae2b131a9e2303b9a5d4b779a3a3b14",
          "datastorepath": "/acs/primary/ref-trl-2240-v-M7-abhishek-kumar/ref-trl-2240-v-M7-abhishek-kumar-esxi-pri2",
          "datastoretype": "NFS",
          "id": "16-2000",
          "imagepath": "[3ae2b131a9e2303b9a5d4b779a3a3b14] import/import_7.vmdk",
          "label": "Hard disk 1",
          "position": 0
        },
        {
          "capacity": 5368709120,
          "controller": "lsilogic",
          "controllerunit": 0,
          "datastorehost": "10.10.0.16",
          "datastorename": "3ae2b131a9e2303b9a5d4b779a3a3b14",
          "datastorepath": "/acs/primary/ref-trl-2240-v-M7-abhishek-kumar/ref-trl-2240-v-M7-abhishek-kumar-esxi-pri2",
          "datastoretype": "NFS",
          "id": "16-2001",
          "imagepath": "[3ae2b131a9e2303b9a5d4b779a3a3b14] import/import_6.vmdk",
          "label": "Hard disk 2",
          "position": 1
        },
        {
          "capacity": 5368709120,
          "controller": "lsilogic",
          "controllerunit": 0,
          "datastorehost": "10.10.0.16",
          "datastorename": "3ae2b131a9e2303b9a5d4b779a3a3b14",
          "datastorepath": "/acs/primary/ref-trl-2240-v-M7-abhishek-kumar/ref-trl-2240-v-M7-abhishek-kumar-esxi-pri2",
          "datastoretype": "NFS",
          "id": "16-2002",
          "imagepath": "[3ae2b131a9e2303b9a5d4b779a3a3b14] import/import_8.vmdk",
          "label": "Hard disk 3",
          "position": 2
        },
        {
          "capacity": 2147483648,
          "controller": "lsilogic",
          "controllerunit": 0,
          "datastorehost": "10.10.0.16",
          "datastorename": "3ae2b131a9e2303b9a5d4b779a3a3b14",
          "datastorepath": "/acs/primary/ref-trl-2240-v-M7-abhishek-kumar/ref-trl-2240-v-M7-abhishek-kumar-esxi-pri2",
          "datastoretype": "NFS",
          "id": "16-2003",
          "imagepath": "[3ae2b131a9e2303b9a5d4b779a3a3b14] import/import_9.vmdk",
          "label": "Hard disk 4",
          "position": 3
        }
      ],
      "hostid": "0029b901-ddf9-4e18-9e3d-244bebc7bd5e",
      "memory": 512,
      "name": "import",
      "nic": [
        {
          "adaptertype": "E1000",
          "id": "Network adapter 1",
          "macaddress": "02:00:1f:66:00:04",
          "networkname": "cloud.guest.1539.200.1-vSwitch1",
          "vlanid": 1539
        }
      ],
      "osdisplayname": "CentOS 4/5 or later (64-bit)",
      "osid": "centos64Guest",
      "powerstate": "PowerOff"
    }
  ]
}

Import attempt without giving data disk mapping:

> import unmanagedinstance clusterid=c4ae9767-cca3-46e3-a801-3218ea8b69d1 name=import serviceofferingid=2564ca56-6ceb-4295-a72b-9c164185584b templateid=8a7a0fe1-5bc0-11eb-89cd-1e00770138e0
{
  "accountid": "a16abb02-5bc0-11eb-89cd-1e00770138e0",
  "cmd": "org.apache.cloudstack.api.command.admin.vm.ImportUnmanagedInstanceCmd",
  "completed": "2021-01-25T07:16:41+0000",
  "created": "2021-01-25T07:16:40+0000",
  "jobid": "559bd155-a641-41ce-a38e-80101cfb6741",
  "jobprocstatus": 0,
  "jobresult": {
    "errorcode": 530,
    "errortext": "VM has total 4 disks for which 0 disk offering mappings provided. 3 disks need a disk offering for import"
  },
  "jobresultcode": 530,
  "jobresulttype": "object",
  "jobstatus": 2,
  "userid": "a16be7b3-5bc0-11eb-89cd-1e00770138e0"
}

Import attempt without giving enough data disk mapping:

> import unmanagedinstance clusterid=c4ae9767-cca3-46e3-a801-3218ea8b69d1 name=import serviceofferingid=2564ca56-6ceb-4295-a72b-9c164185584b templateid=8a7a0fe1-5bc0-11eb-89cd-1e00770138e0 datadiskofferinglist[0].disk=16-2001 datadiskofferinglist[0].diskOffering=c73bbf09-9b34-4010-b735-2fb6a0a930e6
{
  "accountid": "a16abb02-5bc0-11eb-89cd-1e00770138e0",
  "cmd": "org.apache.cloudstack.api.command.admin.vm.ImportUnmanagedInstanceCmd",
  "completed": "2021-01-25T07:16:58+0000",
  "created": "2021-01-25T07:16:58+0000",
  "jobid": "1f15a269-ea02-4ce0-b1cf-0f47d38cb486",
  "jobprocstatus": 0,
  "jobresult": {
    "errorcode": 530,
    "errortext": "VM has total 4 disks for which 1 disk offering mappings provided. 3 disks need a disk offering for import"
  },
  "jobresultcode": 530,
  "jobresulttype": "object",
  "jobstatus": 2,
  "userid": "a16be7b3-5bc0-11eb-89cd-1e00770138e0"
}

Import attempt with wrong data disk ids in mapping but correct mapping count:

 import unmanagedinstance clusterid=c4ae9767-cca3-46e3-a801-3218ea8b69d1 name=import serviceofferingid=2564ca56-6ceb-4295-a72b-9c164185584b templateid=8a7a0fe1-5bc0-11eb-89cd-1e00770138e0 datadiskofferinglist[0].disk=16-2001 datadiskofferinglist[0].diskOffering=c73bbf09-9b34-4010-b735-2fb6a0a930e6 datadiskofferinglist[1].disk=16-2006 datadiskofferinglist[1].diskOffering=c73bbf09-9b34-4010-b735-2fb6a0a930e6 datadiskofferinglist[2].disk=16-2005 datadiskofferinglist[2].diskOffering=c73bbf09-9b34-4010-b735-2fb6a0a930e6
{
  "accountid": "a16abb02-5bc0-11eb-89cd-1e00770138e0",
  "cmd": "org.apache.cloudstack.api.command.admin.vm.ImportUnmanagedInstanceCmd",
  "completed": "2021-01-25T07:17:28+0000",
  "created": "2021-01-25T07:17:28+0000",
  "jobid": "d53495de-4768-4ac3-a9a8-63b1bc34f9d6",
  "jobprocstatus": 0,
  "jobresult": {
    "errorcode": 530,
    "errortext": "VM has total 4 disks, disk offering mapping not provided for 2 disks. Disk IDs that may need a disk offering - 16-2000, 16-2002, 16-2003"
  },
  "jobresultcode": 530,
  "jobresulttype": "object",
  "jobstatus": 2,
  "userid": "a16be7b3-5bc0-11eb-89cd-1e00770138e0"
}

Successful import with correct mapping:

> import unmanagedinstance clusterid=c4ae9767-cca3-46e3-a801-3218ea8b69d1 name=import serviceofferingid=2564ca56-6ceb-4295-a72b-9c164185584b templateid=8a7a0fe1-5bc0-11eb-89cd-1e00770138e0 datadiskofferinglist[0].disk=16-2001 datadiskofferinglist[0].diskOffering=c73bbf09-9b34-4010-b735-2fb6a0a930e6 datadiskofferinglist[1].disk=16-2002 datadiskofferinglist[1].diskOffering=c73bbf09-9b34-4010-b735-2fb6a0a930e6 datadiskofferinglist[2].disk=16-2003 datadiskofferinglist[2].diskOffering=12acfa5b-f4ce-4aca-ab82-dc832c7a9f3e nicnetworklist[0].nic='Network adapter 1' nicnetworklist[0].network=fc3c8122-0a7e-4dd2-ad15-86985fe117a6 nicipaddresslist[0].nic='Network adapter 1' nicipaddresslist[0].ip4Address=auto
{
  "virtualmachine": {
    "account": "admin",
    "affinitygroup": [],
    "cpunumber": 1,
    "cpuspeed": 500,
    "created": "2021-01-25T07:28:48+0000",
    "details": {
      "dataDiskController": "lsilogic",
      "deployvm": "true",
      "nicAdapter": "E1000",
      "rootDiskController": "lsilogic"
    },
    "displayname": "import",
    "displayvm": true,
    "domain": "ROOT",
    "domainid": "8a73a1ac-5bc0-11eb-89cd-1e00770138e0",
    "guestosid": "8a8449f9-5bc0-11eb-89cd-1e00770138e0",
    "haenable": false,
    "hostid": "0029b901-ddf9-4e18-9e3d-244bebc7bd5e",
    "hostname": "10.10.2.159",
    "hypervisor": "VMware",
    "id": "2acdf6dc-a946-4e8a-90c5-c8bd499ff21b",
    "instancename": "import",
    "isdynamicallyscalable": false,
    "memory": 512,
    "name": "import",
    "nic": [
      {
        "extradhcpoption": [],
        "gateway": "10.1.1.1",
        "id": "f65143d5-0cd2-4f71-8bbf-16aea5a585c5",
        "ipaddress": "10.1.1.208",
        "isdefault": true,
        "macaddress": "02:00:1f:66:00:04",
        "netmask": "255.255.255.0",
        "networkid": "fc3c8122-0a7e-4dd2-ad15-86985fe117a6",
        "networkname": "test",
        "secondaryip": [],
        "traffictype": "Guest",
        "type": "Isolated"
      }
    ],
    "osdisplayname": "CentOS 5.3 (64-bit)",
    "ostypeid": "8a8449f9-5bc0-11eb-89cd-1e00770138e0",
    "passwordenabled": false,
    "rootdeviceid": 0,
    "rootdevicetype": "ROOT",
    "securitygroup": [],
    "serviceofferingid": "2564ca56-6ceb-4295-a72b-9c164185584b",
    "serviceofferingname": "Small Instance",
    "state": "Stopped",
    "tags": [],
    "templatedisplaytext": "CentOS 5.3(64-bit) no GUI (vSphere)",
    "templateid": "8a7a0fe1-5bc0-11eb-89cd-1e00770138e0",
    "templatename": "CentOS 5.3(64-bit) no GUI (vSphere)",
    "userid": "a16be7b3-5bc0-11eb-89cd-1e00770138e0",
    "username": "admin",
    "zoneid": "a3932dd2-618b-47de-8d8a-0df2c7e697b9",
    "zonename": "ref-trl-2240-v-M7-abhishek-kumar"
  }
}

shwstppr · 2021-01-18T11:06:14Z

@blueorangutan package

@DK101010 I've drafted this PR to add an optional param for selecting root disk while import

blueorangutan · 2021-01-18T11:07:31Z

@shwstppr a Jenkins job has been kicked to build packages. I'll keep you posted as I make progress.

blueorangutan · 2021-01-18T11:40:31Z

Packaging result: ✔centos7 ✔centos8 ✔debian. JID-2544

rohityadavcloud · 2021-01-18T12:48:05Z

@shwstppr if this is a bugfix, pl change branch to 4.15 and milestone to 4.15.1.0

DK101010 · 2021-01-18T13:18:05Z

@blueorangutan package

@DK101010 I've drafted this PR to add an optional param for selecting root disk while import

@shwstppr Great, for me it looks good. If you want then I can test it in my environment this week.

blueorangutan · 2021-01-18T13:53:00Z

Packaging result: ✔centos7 ✔centos8 ✔debian. JID-2545

blueorangutan · 2021-01-18T13:54:13Z

Packaging result: ✔centos7 ✔centos8 ✔debian. JID-2546

shwstppr · 2021-01-19T11:55:22Z

@rhtyd done

@DK101010 I'm also deliberating of a way to select correct root disk without adding new param in API. For API, we already need to pass a diskid-diskoffering map for all data disks so with that info we can simply select the remaining disk as the root disk (will have to refactor some error messages and change the logic). Does that make sense?

DK101010 · 2021-01-19T13:11:29Z

@rhtyd done

@DK101010 I'm also deliberating of a way to select correct root disk without adding new param in API. For API, we already need to pass a diskid-diskoffering map for all data disks so with that info we can simply select the remaining disk as the root disk (will have to refactor some error messages and change the logic). Does that make sense?

@shwstppr I'm not quite sure if I understand you right. The VCenter give you a list that enables you to find the right root disk?

The diskoffering-map in CS will be created based on the collection sort algo. in my last PR and is wrong in case of that the HardDisk1 is not the root disk. (When I remind me right)

Signed-off-by: Abhishek Kumar <[email protected]>

shwstppr · 2021-01-25T09:27:25Z

@DK101010 I've refactored logic for handling disks and datadiskofferinglist param. As mentioned in the description, correct ROOT disk should be selected now and there won't any need for new API parameters. Will be great if you can test

shwstppr · 2021-01-25T10:16:30Z

@blueorangutan package

blueorangutan · 2021-01-25T10:17:57Z

@shwstppr a Jenkins job has been kicked to build packages. I'll keep you posted as I make progress.

blueorangutan · 2021-01-25T10:55:19Z

Packaging result: ✔centos7 ✖centos8 ✔debian. JID-2585

DK101010 · 2021-01-25T12:00:16Z

@DK101010 I've refactored logic for handling disks and datadiskofferinglist param. As mentioned in the description, correct ROOT disk should be selected now and there won't any need for new API parameters. Will be great if you can test

I'm a little bit skeptical regards the offeringMap :D but I will test it this week.

rohityadavcloud · 2021-01-26T08:56:51Z

@blueorangutan test

blueorangutan · 2021-01-26T08:58:18Z

@rhtyd a Trillian-Jenkins test job (centos7 mgmt + kvm-centos7) has been kicked to run smoke tests

server/src/main/java/org/apache/cloudstack/vm/VmImportManagerImpl.java

blueorangutan · 2021-01-26T20:41:22Z

Trillian test result (tid-3416)
Environment: kvm-centos7 (x2), Advanced Networking with Mgmt server 7
Total time taken: 40480 seconds
Marvin logs: https://github.com/blueorangutan/acs-prs/releases/download/trillian/pr4591-t3416-kvm-centos7.zip
Intermittent failure detected: /marvin/tests/smoke/test_kubernetes_clusters.py
Smoke tests completed. 82 look OK, 1 have error(s)
Only failed tests results shown below:

Test	Result	Time (s)	Test File
test_03_deploy_and_upgrade_kubernetes_cluster	`Failure`	259.54	test_kubernetes_clusters.py
test_04_deploy_and_scale_kubernetes_cluster	`Failure`	4118.10	test_kubernetes_clusters.py

DK101010 · 2021-01-28T12:51:47Z

@shwstppr I've tested it in our environment and it works perfectly.

* 4.14: server: select root disk based on user input during vm import (#4591) kvm: Use Q35 chipset for UEFI x86_64 (#4576) server: fix wrong error message when create isolated network without SourceNat (#4624) server: add possibility to scale vm to current customer offerings (#4622) server: keep networks order and ips while move a vm with multiple networks (#4602) server: throw exception when update vm nic on L2 network (#4625) doc: fix typo in install notes (#4633)

* 4.15: server: select root disk based on user input during vm import (#4591) kvm: Use Q35 chipset for UEFI x86_64 (#4576) server: fix wrong error message when create isolated network without SourceNat (#4624) server: add possibility to scale vm to current customer offerings (#4622) server: keep networks order and ips while move a vm with multiple networks (#4602) server: throw exception when update vm nic on L2 network (#4625) doc: fix typo in install notes (#4633)

shwstppr self-assigned this Jan 18, 2021

shwstppr mentioned this pull request Jan 18, 2021

wrong root disk during the ingest bug #4547

Closed

shwstppr linked an issue Jan 18, 2021 that may be closed by this pull request

wrong root disk during the ingest bug #4547

Closed

shwstppr added the status:wip label Jan 18, 2021

shwstppr force-pushed the add-vm-import-rootdisk-param branch from 4240c54 to 9d532af Compare January 19, 2021 11:47

shwstppr changed the base branch from master to 4.15 January 19, 2021 11:47

server: select root disk based on user input during vm import

020c42a

Signed-off-by: Abhishek Kumar <[email protected]>

shwstppr force-pushed the add-vm-import-rootdisk-param branch from 9d532af to 020c42a Compare January 25, 2021 07:32

shwstppr changed the base branch from 4.15 to 4.14 January 25, 2021 07:33

shwstppr changed the title ~~api, server: add rootdiskid param in importUnmanagedInstance API~~ server: select root disk based on user input during vm import Jan 25, 2021

shwstppr requested review from andrijapanicsb and nvazquez January 25, 2021 09:24

shwstppr marked this pull request as ready for review January 25, 2021 09:25

shwstppr added component:vmware type:bug labels Jan 25, 2021

shwstppr added this to the 4.14.1.0 milestone Jan 25, 2021

shwstppr removed the status:wip label Jan 25, 2021

shwstppr mentioned this pull request Jan 25, 2021

Bug/disk order bug during ingest #4548

Closed

12 tasks

apache deleted a comment from blueorangutan Jan 25, 2021

sureshanaparti reviewed Jan 26, 2021

View reviewed changes

server/src/main/java/org/apache/cloudstack/vm/VmImportManagerImpl.java Show resolved Hide resolved

sureshanaparti reviewed Jan 26, 2021

View reviewed changes

server/src/main/java/org/apache/cloudstack/vm/VmImportManagerImpl.java Show resolved Hide resolved

rohityadavcloud approved these changes Feb 1, 2021

View reviewed changes

sureshanaparti approved these changes Feb 1, 2021

View reviewed changes

DaanHoogland merged commit 9b45ec2 into apache:4.14 Feb 1, 2021

DaanHoogland deleted the add-vm-import-rootdisk-param branch February 1, 2021 09:55

server: select root disk based on user input during vm import #4591

server: select root disk based on user input during vm import #4591

Uh oh!

Conversation

shwstppr commented Jan 18, 2021 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Description

Types of changes

Feature/Enhancement Scale or Bug Severity

Feature/Enhancement Scale

Bug Severity

Screenshots (if appropriate):

How Has This Been Tested?

Uh oh!

shwstppr commented Jan 18, 2021

Uh oh!

blueorangutan commented Jan 18, 2021

Uh oh!

blueorangutan commented Jan 18, 2021

Uh oh!

rohityadavcloud commented Jan 18, 2021

Uh oh!

DK101010 commented Jan 18, 2021

Uh oh!

blueorangutan commented Jan 18, 2021

Uh oh!

blueorangutan commented Jan 18, 2021

Uh oh!

shwstppr commented Jan 19, 2021

Uh oh!

DK101010 commented Jan 19, 2021 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

shwstppr commented Jan 25, 2021

Uh oh!

shwstppr commented Jan 25, 2021

Uh oh!

blueorangutan commented Jan 25, 2021

Uh oh!

blueorangutan commented Jan 25, 2021

Uh oh!

DK101010 commented Jan 25, 2021

Uh oh!

rohityadavcloud commented Jan 26, 2021

Uh oh!

blueorangutan commented Jan 26, 2021

Uh oh!

Uh oh!

Uh oh!

blueorangutan commented Jan 26, 2021

Uh oh!

DK101010 commented Jan 28, 2021

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

6 participants

shwstppr commented Jan 18, 2021 •

edited

Loading

DK101010 commented Jan 19, 2021 •

edited

Loading