Can't remove a failed ISO

[tampler]

ISSUE TYPE

• Bug Report

COMPONENT NAME

UI, API

CLOUDSTACK VERSION

4.15.1

CONFIGURATION

Basic network, one bridge, one zone

OS / ENVIRONMENT

Ubuntu 20.04.2
cmk 6.1.0

SUMMARY

1. After failing to upload an ISO from the local webserver, CS shows it in the list of ISOs with the status "Ready : false". But I can't remove it from both GUI and cmk

2. cmk output:

Apache CloudStack 🐵 CloudMonkey 6.1.0
Report issues: https://github.com/apache/cloudstack-cloudmonkey/issues

(localcloud) 🐱 > list isos 
(localcloud) 🐱 > delete iso id=d4fb61a3-f5e1-4026-a59e-cfb8279ee453
{
  "accountid": "6c1b9fb9-ea09-11eb-837e-0015179e5d77",
  "cmd": "org.apache.cloudstack.api.command.user.iso.DeleteIsoCmd",
  "completed": "2021-07-21T23:29:24+0300",
  "created": "2021-07-21T23:29:22+0300",
  "jobid": "a710ccf4-10fb-4465-ae3f-aa4a30162c23",
  "jobinstanceid": "d4fb61a3-f5e1-4026-a59e-cfb8279ee453",
  "jobinstancetype": "Iso",
  "jobprocstatus": 0,
  "jobresult": {
    "errorcode": 530,
    "errortext": "Failed to delete ISO"
  },
  "jobresultcode": 530,
  "jobresulttype": "object",
  "jobstatus": 2,
  "userid": "6c1f585f-ea09-11eb-837e-0015179e5d77"
}
🙈 Error: async API failed for job a710ccf4-10fb-4465-ae3f-aa4a30162c23

STEPS TO REPRODUCE

EXPECTED RESULTS

Expect to be able to add/remove ISOs

ACTUAL RESULTS

Failed to add an ISO. Failed to remove a bogus ISO

[Pearl1594]

@tampler Is your setup/env SSL enabled, if not, uploading an iso from local will fail. With respect to failure in deleting an ISO, I wasn't able to reproduce the issue. Can you check on updating the following global setting configs:storage.cleanup.delayand storage.cleanup.interval to a low value, does the iso get cleaned up automatically after the configured time has elapsed. That worked for me. You can also check the log file at /var/log/cloudstack/management/management-server.log for hints as to why the failure occurred during cleanup. Thanks

[weizhouapache]

@tampler ssl is required in SSVM when upload template/iso form local, see @Pearl1594 's comment above
you may find information on https://www.shapeblue.com/securing-cloudstack-4-11-with-https-tls/

the issue with remove failed ISO should has been fixed by #4626 in cloudstack 4.15.1.0
if you faced the issue in 4.15.1.0 testing (not 4.15.0.0), please share more logs .

[tampler]

I see the same issue for Management Servers. I got only one management server, but see three with different IDs in the GUI and can't remove any of them as well

Should I open a new bug report for this ?

[tampler]

@ustcweizhou I'm not sure which logs do you want. Here's an exert from the /var/logs/cloudstack/management/management-server.log. See the file attached.

Let me know if you need an output from some specific cmk command.
out.log

Thank you for your help

[weizhouapache]

I see the same issue for Management Servers. I got only one management server, but see three with different IDs in the GUI and can't remove any of them as well

@tampler it is not a bug I think.
could you share table 'mshost' in database ?

mysql> select * from mshost;

[weizhouapache]

@ustcweizhou I'm not sure which logs do you want. Here's an exert from the /var/logs/cloudstack/management/management-server.log. See the file attached.

Let me know if you need an output from some specific cmk command.
out.log

Thank you for your help

@tampler could you please check if SSVM (secondary storage vm) is running and its Agent state is Up ?
Infrastructure -> System VMs -> s-X-VM

[tampler]

I see the same issue for Management Servers. I got only one management server, but see three with different IDs in the GUI and can't remove any of them as well

@tampler it is not a bug I think.
could you share table 'mshost' in database ?

mysql> select * from mshost;

Sure thing, here's the screen:

[tampler]

@tampler could you please check if SSVM (secondary storage vm) is running and its Agent state is Up ?
Infrastructure -> System VMs -> s-X-VM

No, they are NOT. I can't launch any VM and haven't seen SystemVMs working due to #5235

However, I still see those ISOs in GUI and can't remove them. I also tried official Ubuntu image. Its still won't get to the Ready: true state and can't be removed

[nvazquez]

@tampler to fix this you will need your SSVM deployed, looks like an issue in your setup is causing the system VMs to fail

[tampler]

Seems like a Zone setup issue. Can't reproduce, closing for now