Add local ISO upload via UI #3251

nvazquez · 2019-04-01T11:43:51Z

Description

Problem: Users can register ISOs from URL but cannot upload local ISOs.

Root cause: CloudStack provides browser-based upload support for volumes and templates, but ISOs are not supported.

Solution:
The existing browser-based upload from local functionality for templates and volumes (https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=39620237) is extended to support uploading local ISOs.

Extend the UI: A new button is created under the ISOs view: 'Upload from Local'. A new dialog form is displayed in which the user must select the ISO to upload from its local file system.
Extend the API: New 'GetUploadParamsForIso' API command is created to handle the ISO upload.

Types of changes

Breaking change (fix or feature that would cause existing functionality to change)
New feature (non-breaking change which adds functionality)
Bug fix (non-breaking change which fixes an issue)
Enhancement (improves an existing feature and functionality)
Cleanup (Code refactoring and cleanup, that may add test cases)

Screenshots (if appropriate):

How Has This Been Tested?

Templates tab -> ISOs -> Upload from local -> Select file on local file system

DaanHoogland · 2019-04-01T12:20:40Z

@blueorangutan package

blueorangutan · 2019-04-01T12:21:24Z

@DaanHoogland a Jenkins job has been kicked to build packages. I'll keep you posted as I make progress.

blueorangutan · 2019-04-01T12:41:59Z

Packaging result: ✔centos6 ✔centos7 ✔debian. JID-2666

DaanHoogland

looks ok, in spite of my comments. awaiting testing and such

api/src/main/java/org/apache/cloudstack/api/command/user/iso/GetUploadParamsForIsoCmd.java

server/src/main/java/com/cloud/template/TemplateAdapterBase.java

borisstoyanov · 2019-04-02T12:09:19Z

@blueorangutan package

blueorangutan · 2019-04-02T12:09:29Z

@borisstoyanov a Jenkins job has been kicked to build packages. I'll keep you posted as I make progress.

blueorangutan · 2019-04-02T12:30:10Z

Packaging result: ✔centos6 ✖centos7 ✔debian. JID-2669

nvazquez · 2019-04-02T12:34:41Z

@blueorangutan package

blueorangutan · 2019-04-02T12:35:29Z

@nvazquez a Jenkins job has been kicked to build packages. I'll keep you posted as I make progress.

blueorangutan · 2019-04-02T12:56:38Z

Packaging result: ✔centos6 ✔centos7 ✔debian. JID-2670

nvazquez · 2019-04-02T18:06:50Z

@blueorangutan test

blueorangutan · 2019-04-02T18:07:30Z

@nvazquez a Trillian-Jenkins test job (centos7 mgmt + kvm-centos7) has been kicked to run smoke tests

blueorangutan · 2019-04-03T03:10:44Z

Trillian test result (tid-3463)
Environment: kvm-centos7 (x2), Advanced Networking with Mgmt server 7
Total time taken: 30763 seconds
Marvin logs: https://github.com/blueorangutan/acs-prs/releases/download/trillian/pr3251-t3463-kvm-centos7.zip
Intermittent failure detected: /marvin/tests/smoke/test_privategw_acl.py
Smoke tests completed. 69 look OK, 1 have error(s)
Only failed tests results shown below:

Test	Result	Time (s)	Test File
test_04_rvpc_privategw_static_routes	`Failure`	938.94	test_privategw_acl.py

borisstoyanov

LGTM, manually checked the upload is working

nvazquez · 2019-04-04T11:18:50Z

@blueorangutan package

blueorangutan · 2019-04-04T11:19:38Z

@nvazquez a Jenkins job has been kicked to build packages. I'll keep you posted as I make progress.

blueorangutan · 2019-04-04T11:40:56Z

Packaging result: ✔centos6 ✔centos7 ✔debian. JID-2679

nvazquez · 2019-04-04T11:48:30Z

@blueorangutan test

blueorangutan · 2019-04-04T11:48:38Z

@nvazquez a Trillian-Jenkins test job (centos7 mgmt + kvm-centos7) has been kicked to run smoke tests

PaulAngus · 2019-04-08T16:46:43Z

I just tried to upload an iso from my local machine and from a UI perspective it failed silently, after 20 mins it appeared to still be uploading.

However from the logs, there was an immediate NPE:

2019-04-08 16:37:17,578 DEBUG [c.c.a.ApiServlet] (qtp504527234-16:ctx-889f9019) (logid:cb39b7a8) ===START===  10.0.0.20 -- GET  command=getUploadParamsForIso&response=json&name=freenas&displayText=freenastest&zoneid=ceaaa03a-d986-4187-941a-fdc45ff51892&format=ISO&isextractable=false&bootable=true&ispublic=true&isfeatured=false&osTypeId=3e7a9391-5a1a-11e9-a4f6-1e0081010813&_=1554741208165
2019-04-08 16:37:17,581 DEBUG [c.c.a.ApiServer] (qtp504527234-16:ctx-889f9019 ctx-8af4a41b) (logid:cb39b7a8) CIDRs from which account 'Acct[5fe0af81-5a1a-11e9-a4f6-1e0081010813-admin]' is allowed to perform API calls: 0.0.0.0/0,::/0
2019-04-08 16:37:17,596 DEBUG [c.c.u.d.T.Transaction] (qtp504527234-16:ctx-889f9019 ctx-8af4a41b) (logid:cb39b7a8) Rolling back the transaction: Time = 0 Name =  qtp504527234-16; called by -TransactionLegacy.rollback:890-TransactionLegacy.removeUpTo:833-TransactionLegacy.close:657-Transaction.execute:43-Transaction.execute:47-HypervisorTemplateAdapter.createTemplateForPostUpload:292-TemplateManagerImpl.registerPostUploadInternal:361-TemplateManagerImpl.registerIsoForPostUpload:415-NativeMethodAccessorImpl.invoke0:-2-NativeMethodAccessorImpl.invoke:62-DelegatingMethodAccessorImpl.invoke:43-Method.invoke:498
2019-04-08 16:37:17,600 ERROR [c.c.a.ApiServer] (qtp504527234-16:ctx-889f9019 ctx-8af4a41b) (logid:cb39b7a8) unhandled exception executing api command: [Ljava.lang.String;@287808f9
java.lang.NullPointerException
	at com.cloud.template.TemplateAdapterBase.persistTemplate(TemplateAdapterBase.java:368)
	at com.cloud.template.HypervisorTemplateAdapter$1.doInTransaction(HypervisorTemplateAdapter.java:297)
	at com.cloud.template.HypervisorTemplateAdapter$1.doInTransaction(HypervisorTemplateAdapter.java:292)
	at com.cloud.utils.db.Transaction$2.doInTransaction(Transaction.java:50)
	at com.cloud.utils.db.Transaction.execute(Transaction.java:40)
	at com.cloud.utils.db.Transaction.execute(Transaction.java:47)
	at com.cloud.template.HypervisorTemplateAdapter.createTemplateForPostUpload(HypervisorTemplateAdapter.java:292)
	at com.cloud.template.TemplateManagerImpl.registerPostUploadInternal(TemplateManagerImpl.java:361)
	at com.cloud.template.TemplateManagerImpl.registerIsoForPostUpload(TemplateManagerImpl.java:415)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:338)
	at org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:197)
	at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:163)
	at org.apache.cloudstack.network.contrail.management.EventUtils$EventInterceptor.invoke(EventUtils.java:107)
	at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:174)
	at com.cloud.event.ActionEventInterceptor.invoke(ActionEventInterceptor.java:51)
	at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:174)
	at org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:92)
	at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:185)
	at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:212)
	at com.sun.proxy.$Proxy182.registerIsoForPostUpload(Unknown Source)
	at org.apache.cloudstack.api.command.user.iso.GetUploadParamsForIsoCmd.execute(GetUploadParamsForIsoCmd.java:130)
	at com.cloud.api.ApiDispatcher.dispatch(ApiDispatcher.java:156)
	at com.cloud.api.ApiServer.queueCommand(ApiServer.java:758)
	at com.cloud.api.ApiServer.handleRequest(ApiServer.java:582)
	at com.cloud.api.ApiServlet.processRequestInContext(ApiServlet.java:310)
	at com.cloud.api.ApiServlet$1.run(ApiServlet.java:130)
	at org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:56)
	at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:103)
	at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:53)
	at com.cloud.api.ApiServlet.processRequest(ApiServlet.java:127)
	at com.cloud.api.ApiServlet.doGet(ApiServlet.java:89)
	at javax.servlet.http.HttpServlet.service(HttpServlet.java:686)
	at javax.servlet.http.HttpServlet.service(HttpServlet.java:791)
	at org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:852)
	at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:535)
	at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:143)
	at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:548)
	at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:132)
	at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:190)
	at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:1595)
	at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:188)
	at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1253)
	at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:168)
	at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:473)
	at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:1564)
	at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:166)
	at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1155)
	at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)
	at org.eclipse.jetty.server.handler.gzip.GzipHandler.handle(GzipHandler.java:527)
	at org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:126)
	at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:132)
	at org.eclipse.jetty.server.Server.handle(Server.java:530)
	at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:347)
	at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:256)
	at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:279)
	at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:102)
	at org.eclipse.jetty.io.ChannelEndPoint$2.run(ChannelEndPoint.java:124)
	at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:247)
	at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.produce(EatWhatYouKill.java:140)
	at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.run(EatWhatYouKill.java:131)
	at org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run(ReservedThreadExecutor.java:382)
	at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:708)
	at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:626)
	at java.lang.Thread.run(Thread.java:748)

@nvazquez @borisstoyanov

PaulAngus · 2019-04-15T13:14:10Z

Not sure that this message should go to end users. This is good for logs. but end users should get something LESS specific.

blueorangutan · 2019-04-15T19:28:05Z

Trillian test result (tid-3495)
Environment: kvm-centos7 (x2), Advanced Networking with Mgmt server 7
Total time taken: 24603 seconds
Marvin logs: https://github.com/blueorangutan/acs-prs/releases/download/trillian/pr3251-t3495-kvm-centos7.zip
Intermittent failure detected: /marvin/tests/smoke/test_hostha_kvm.py
Smoke tests completed. 69 look OK, 1 have error(s)
Only failed tests results shown below:

Test	Result	Time (s)	Test File
test_hostha_enable_ha_when_host_in_maintenance	`Error`	302.62	test_hostha_kvm.py

anuragaw · 2019-04-17T08:16:55Z

Verified @borisstoyanov observation that when create button is clicked for templates or ISOs an entry is created in database and even in failed state a wrong entry is created. Logic seems to be separable but needs ~1 day for separation and testing.

For message can we show something more generic like - "Error in uploading ISO. Please check system settings on SSVM." ?

CC/ @PaulAngus , @rhtyd , @borisstoyanov

yadvr · 2019-04-17T08:20:21Z

@anuragaw I would prefer something like "Failed to upload {iso/template} due to system configuration please contact your administrator." and in the API that finds the URL of the remote endpoint simply check if we've setup the ssl certificate (from keystore table we can check if cpvm/ssvm certificate exists) and log a statement that upload will possibly fail due to missing SSL cert setup.

yadvr · 2019-04-17T10:40:55Z

The feature of uploading a volume/template and with this PR iso requires that the env has ssl/tls certificates setup. The failure to upload a template/volume/iso when endpoint is incorrectly setup with a wrong certificate (default) is a known case and should not limit the acceptance of the PR. Instead in a separate PR we can attempt to allow users to upload a template/volume/iso when admin has not setup a certs by providing it a non-https endpoint of the ssvm.

ui/scripts/ui/dialog.js

yadvr · 2019-05-23T14:28:46Z

@blueorangutan package

blueorangutan · 2019-05-23T14:28:59Z

@rhtyd a Jenkins job has been kicked to build packages. I'll keep you posted as I make progress.

blueorangutan · 2019-05-23T16:11:15Z

Packaging result: ✔centos6 ✔centos7 ✔debian. JID-2760

borisstoyanov

LGTM

nvazquez · 2019-05-30T18:25:49Z

Thanks @borisstoyanov @rhtyd, I'll close/open to re-kick Travis

anuragaw

LGTM based on code and local testing.

PaulAngus

LGTM - tested by uploading a VyOS iso from local machine and using it to create a new instance.

yadvr · 2019-06-05T17:40:16Z

Merging based on last smoketests and reviews.

nvazquez added the status:work-in-progress label Apr 1, 2019

nvazquez added this to the 4.13.0.0 milestone Apr 1, 2019

nvazquez self-assigned this Apr 1, 2019

nvazquez requested review from PaulAngus and borisstoyanov April 1, 2019 11:43

nvazquez mentioned this pull request Apr 1, 2019

Add local ISO upload via UI shapeblue/cloudstack#35

Closed

5 tasks

DaanHoogland reviewed Apr 1, 2019

View reviewed changes

api/src/main/java/org/apache/cloudstack/api/command/user/iso/GetUploadParamsForIsoCmd.java Outdated Show resolved Hide resolved

server/src/main/java/com/cloud/template/TemplateAdapterBase.java Outdated Show resolved Hide resolved

nvazquez added 2 commits April 1, 2019 10:26

Add local ISO upload via UI

498711a

Fix API since value

dcb9875

nvazquez force-pushed the addlocalisoupload branch from 24750aa to dcb9875 Compare April 2, 2019 01:03

borisstoyanov approved these changes Apr 3, 2019

View reviewed changes

Refactor upload parameters

1ef1d81

anuragaw reviewed Apr 18, 2019

View reviewed changes

ui/scripts/ui/dialog.js Outdated Show resolved Hide resolved

Make error message more generic as it's in dialog.js

6cf163d

borisstoyanov reviewed Apr 18, 2019

View reviewed changes

ui/scripts/ui/dialog.js Show resolved Hide resolved

borisstoyanov approved these changes May 28, 2019

View reviewed changes

nvazquez changed the title ~~[WIP DO NOT MERGE] Add local ISO upload via UI~~ Add local ISO upload via UI May 29, 2019

nvazquez removed the status:work-in-progress label May 29, 2019

yadvr approved these changes May 30, 2019

View reviewed changes

nvazquez closed this May 30, 2019

nvazquez reopened this May 30, 2019

nvazquez closed this May 31, 2019

nvazquez reopened this May 31, 2019

nvazquez closed this Jun 3, 2019

nvazquez reopened this Jun 3, 2019

anuragaw approved these changes Jun 4, 2019

View reviewed changes

PaulAngus approved these changes Jun 5, 2019

View reviewed changes

yadvr approved these changes Jun 5, 2019

View reviewed changes

yadvr merged commit 7247c5e into apache:master Jun 5, 2019

ustcweizhou mentioned this pull request Jul 16, 2019

Allow ISO Upload from Local Machine #3496

Closed

nvazquez deleted the addlocalisoupload branch April 6, 2020 14:42

nlgordon mentioned this pull request Mar 15, 2023

AC-135 Allow Upload of ISOs ippathways/cloudstack#72

Merged

Add local ISO upload via UI #3251

Add local ISO upload via UI #3251

Uh oh!

Conversation

nvazquez commented Apr 1, 2019

Description

Types of changes

Screenshots (if appropriate):

How Has This Been Tested?

Uh oh!

DaanHoogland commented Apr 1, 2019

Uh oh!

blueorangutan commented Apr 1, 2019

Uh oh!

blueorangutan commented Apr 1, 2019

Uh oh!

DaanHoogland left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

borisstoyanov commented Apr 2, 2019

Uh oh!

blueorangutan commented Apr 2, 2019

Uh oh!

blueorangutan commented Apr 2, 2019

Uh oh!

nvazquez commented Apr 2, 2019

Uh oh!

blueorangutan commented Apr 2, 2019

Uh oh!

blueorangutan commented Apr 2, 2019

Uh oh!

nvazquez commented Apr 2, 2019

Uh oh!

blueorangutan commented Apr 2, 2019

Uh oh!

blueorangutan commented Apr 3, 2019

Uh oh!

borisstoyanov left a comment

Choose a reason for hiding this comment

Uh oh!

nvazquez commented Apr 4, 2019

Uh oh!

blueorangutan commented Apr 4, 2019

Uh oh!

blueorangutan commented Apr 4, 2019

Uh oh!

nvazquez commented Apr 4, 2019

Uh oh!

blueorangutan commented Apr 4, 2019

Uh oh!

PaulAngus commented Apr 8, 2019

Uh oh!

PaulAngus commented Apr 15, 2019

Uh oh!

blueorangutan commented Apr 15, 2019

Uh oh!

anuragaw commented Apr 17, 2019

Uh oh!

yadvr commented Apr 17, 2019

Uh oh!

yadvr commented Apr 17, 2019

Uh oh!

Uh oh!

Uh oh!

yadvr commented May 23, 2019

Uh oh!

blueorangutan commented May 23, 2019

Uh oh!

blueorangutan commented May 23, 2019

Uh oh!

borisstoyanov left a comment

Choose a reason for hiding this comment

Uh oh!

nvazquez commented May 30, 2019

Uh oh!

anuragaw left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

anuragaw left a comment •

edited

Loading