Unable to create a deployment for VM instance 4.19

[tamoorahmedntu]

ISSUE TYPE

• Other

COMPONENT NAME

SSVM
CPVM

CLOUDSTACK VERSION

CloudStack 4.19.1.0 

CONFIGURATION

Advance Networking
Security Group 

OS / ENVIRONMENT

Ubuntu 22.04
KVM

1. Managment 
2. Host M
3. Host S
4. Storage Server NFS

SUMMARY

WARN  [o.a.c.s.SecondaryStorageManagerImpl] (API-Job-Executor-27:ctx-39a91335 job-97 ctx-a2fc4c8c) (logid:f672b9aa) Unable to start secondary storage VM [8] due to [Unable to create a deployment for VM instance {"id":8,"instanceName":"s-8-VM","type":"SecondaryStorageVm","uuid":"968e3eb3-fdee-482b-b593-961c5f37d973"}].
com.cloud.exception.InsufficientServerCapacityException: Unable to create a deployment for VM instance {"id":8,"instanceName":"s-8-VM","type":"SecondaryStorageVm","uuid":"968e3eb3-fdee-482b-b593-961c5f37d973"}Scope=interface com.cloud.dc.DataCenter; id=1
	at com.cloud.vm.VirtualMachineManagerImpl.orchestrateStart(VirtualMachineManagerImpl.java:1228)
	at com.cloud.vm.VirtualMachineManagerImpl.orchestrateStart(VirtualMachineManagerImpl.java:5486)
	at jdk.internal.reflect.GeneratedMethodAccessor395.invoke(Unknown Source)
	at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.base/java.lang.reflect.Method.invoke(Method.java:566)
	at com.cloud.vm.VmWorkJobHandlerProxy.handleVmWorkJob(VmWorkJobHandlerProxy.java:105)
	at com.cloud.vm.VirtualMachineManagerImpl.handleVmWorkJob(VirtualMachineManagerImpl.java:5610)
	at com.cloud.vm.VmWorkJobDispatcher.runJob(VmWorkJobDispatcher.java:102)
	at org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.runInContext(AsyncJobManagerImpl.java:654)
	at org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedContextRunnable.java:48)
	at org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:55)
	at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:102)
	at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:52)
	at org.apache.cloudstack.managed.context.ManagedContextRunnable.run(ManagedContextRunnable.java:45)
	at org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.run(AsyncJobManagerImpl.java:602)
	at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
	at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
	at java.base/java.lang.Thread.run(Thread.java:829)

STEPS TO REPRODUCE

Setup New Advance Networking with new Zone on Ubuntu 22.04 Using KVM 

EXPECTED RESULTS

SSVM Start Up
CPVM Start Up

ACTUAL RESULTS

SSVM Doesn't Up
CPVM Doesn't Up

Logs attached Below:

management-server.log
agent.log

network:
  version: 2
  ethernets:
    ens160:
      dhcp4: false
    ens33:
      dhcp4: false
  bridges:
   cloudbr0:
    addresses: [0.0.168.2/23]
    routes:
     - to: default
       via: 0.0.168.254
    nameservers:
     addresses: [0.0.0.0,0.0.0.0]
    interfaces: [ens160]
    dhcp4: false
    dhcp6: false
    parameters:
       stp: false
       forward-delay: 0
   cloudbr1:
    dhcp4: false
    interfaces: [ens33]
    parameters:
       stp: false

[rajujith]

@tamoorahmedntu Based on the logs you you have shared, the SSVM was deployed successfully but it got destroyed later on? the job in snippets you pasted indicate either the host is in 'Alert', disabled or disconnected. It is not finding a suitable host for deploying the SSVM.

[tamoorahmedntu]

This is the issue im facing the hosts and up and running with everything installed but for what ever reason the ssvm and CPVM can not run on the hosts it keeps failing and then destroying then creating another one. The host has everything installed and is correct.

[rajujith]

@tamoorahmedntu Here are a couple of more checks you can do:

1. Make sure the KVM hosts are Up and enabled
2. Make sure the cluster is Enabled and Managed.
3. Restart cloudstack-management service to test if the situation improves.

[tamoorahmedntu]

I have done this and the they still don't seem to come up,
I have double checked the kvm and the requirements, the system was working before but after a reboot. things stopped working

[weizhouapache]

@tamoorahmedntu
is the host Ip changed after reboot ?

[tamoorahmedntu]

no everything is static

[weizhouapache]

no everything is static

can you share the output of ip a and ip route commands ?

the ip addresses in netplan configuration looks not normal.

[tamoorahmedntu]

Management :

Host :

2nd Host :

[xmlgrg]

I deployed 4.19 on an Ubuntu 22.04 and encountered the same problem as you

[Revellski]

• Make sure the secondary storage can be mounted on the KVM nodes.
• Your hosts are not in alert state
• Zone, pod and cluster are all Enabled

[tamoorahmedntu]

i used virsh-pool info and used mount | grep nfs which both showed drive mounted i also mounted manually and wrote a file in each case
the hosts are green
zone is disabled because if i dont cloudstack keeps destroying and recreating and while fixing this issue the numbers keep going up

[tamoorahmedntu]

I had to wipe everything and rebuild from scratch everything