backports for 4.11.1 from master (#2621)

* CLOUDSTACK-10147 Disabled Xenserver Cluster can still deploy VM's. Added code to skip disabled clusters when selecting a host (#2442)

(cherry picked from commit c3488a5)
Signed-off-by: Rohit Yadav <[email protected]>

* CLOUDSTACK-10318: Bug on sorting ACL rules list in chrome (#2478)

(cherry picked from commit 4412563)
Signed-off-by: Rohit Yadav <[email protected]>

* CLOUDSTACK-10284:Creating a snapshot from VM Snapshot generates error if hypervisor is not KVM.

Signed-off-by: Rohit Yadav <[email protected]>

* CLOUDSTACK-10221: Allow IPv6 when creating a Basic Network (#2397)

Since CloudStack 4.10 Basic Networking supports IPv6 and thus
should be allowed to be specified when creating a network.

Signed-off-by: Wido den Hollander <[email protected]>
(cherry picked from commit 9733a10)
Signed-off-by: Rohit Yadav <[email protected]>

* CLOUDSTACK-10214: Unable to remove local primary storage (#2390)

Allow admins to remove primary storage pool.
Cherry-picked from eba2e1d

Signed-off-by: Rohit Yadav <[email protected]>

* dateutil: constistency of tzdate input and output (#2392)

Signed-off-by: Yoan Blanc <[email protected]>
Signed-off-by: Daan Hoogland <[email protected]>
(cherry picked from commit 2ad5202)
Signed-off-by: Rohit Yadav <[email protected]>

* CLOUDSTACK-10054:Volume download times out in 3600 seconds (#2244)

(cherry picked from commit bb607d0)
Signed-off-by: Rohit Yadav <[email protected]>

* When creating a new account (via domain admin) it is possible to select “root admin” as the role for the new user (#2606)

* create account with domain admin showing 'root admin' role

Domain admins should not be able to assign the role of root admin to new users. Therefore, the role ‘root admin’ (or any other of the same type) should not be visible to domain admins.

* License and formatting

* Break long sentence into multiple lines

* Fix wording of method 'getCurrentAccount'

* fix typo in variable name

* [CLOUDSTACK-10259] Missing float part of secondary storage data in listAccounts

* [CLOUDSTACK-9338] ACS not accounting resources of VMs with custom service offering

ACS is accounting the resources properly when deploying VMs with custom service offerings. However, there are other methods (such as updateResourceCount) that do not execute the resource accounting properly, and these methods update the resource count for an account in the database. Therefore, if a user deploys VMs with custom service offerings, and later this user calls the “updateResourceCount” method, it (the method) will only account for VMs with normal service offerings, and update this as the number of resources used by the account. This will result in a smaller number of resources to be accounted for the given account than the real used value. The problem becomes worse because if the user starts to delete these VMs, it is possible to reach negative values of resources allocated (breaking all of the resource limiting for accounts). This is a very serious attack vector for public cloud providers!

* [CLOUDSTACK-10230] User should not be able to use removed “Guest OS type” (#2404)

* [CLOUDSTACK-10230] User is able to change to “Guest OS type” that has been removed

Users are able to change the OS type of VMs to “Guest OS type” that has been removed. This becomes a security issue when we try to force users to use HVM VMs (Meltdown/Spectre thing). A removed “guest os type” should not be usable by any users in the cloud.

* Remove trailing lines that are breaking build due to checkstyle compliance

* Remove unused imports

* fix classes that were in the wrong folder structure

* Updates to capacity management

Author: yadvr

api/src/com/cloud/configuration/Resource.java

@@ -38,8 +38,8 @@ public enum ResourceType { // Primary and Secondary storage are allocated_storag
         private ResourceOwnerType[] supportedOwners;
         private int ordinal;
         public static final long bytesToKiB = 1024;
-        public static final long bytesToMiB = 1024 * 1024;
-        public static final long bytesToGiB = 1024 * 1024 * 1024;
+        public static final long bytesToMiB = bytesToKiB * 1024;
+        public static final long bytesToGiB = bytesToMiB * 1024;
 
         ResourceType(String name, int ordinal, ResourceOwnerType... supportedOwners) {
             this.name = name;

…ception/UnavailableCommandException.java …ception/UnavailableCommandException.javaapi/src/main/java/com/cloud/exception/UnavailableCommandException.java renamed to api/src/com/cloud/exception/UnavailableCommandException.java api/src/main/java/com/cloud/exception/UnavailableCommandException.java renamed to api/src/com/cloud/exception/UnavailableCommandException.java

@@ -17,38 +17,64 @@
 
 package org.apache.cloudstack.acl;
 
+import java.util.List;
+
 import org.apache.cloudstack.acl.RolePermission.Permission;
 import org.apache.cloudstack.framework.config.ConfigKey;
 
-import java.util.List;
-
 public interface RoleService {
 
     ConfigKey<Boolean> EnableDynamicApiChecker = new ConfigKey<>("Advanced", Boolean.class, "dynamic.apichecker.enabled", "false",
-            "If set to true, this enables the dynamic role-based api access checker and disables the default static role-based api access checker.",
-            true);
+            "If set to true, this enables the dynamic role-based api access checker and disables the default static role-based api access checker.", true);
 
     boolean isEnabled();
-    Role findRole(final Long id);
-    Role createRole(final String name, final RoleType roleType, final String description);
-    Role updateRole(final Role role, final String name, final RoleType roleType, final String description);
-    boolean deleteRole(final Role role);
 
-    RolePermission findRolePermission(final Long id);
-    RolePermission findRolePermissionByUuid(final String uuid);
+    /**
+     *  Searches for a role with the given ID. If the ID is null or less than zero, this method will return null.
+     *  This method will also return null if no role is found with the provided ID.
+     *  Moreover, we will check if the requested role is of 'Admin' type; roles with 'Admin' type should only be visible to 'root admins'.
+     *  Therefore, if a non-'root admin' user tries to search for an 'Admin' role, this method will return null.
+     */
+    Role findRole(Long id);
+
+    Role createRole(String name, RoleType roleType, String description);
+
+    Role updateRole(Role role, String name, RoleType roleType, String description);
+
+    boolean deleteRole(Role role);
+
+    RolePermission findRolePermission(Long id);
+
+    RolePermission findRolePermissionByUuid(String uuid);
+
+    RolePermission createRolePermission(Role role, Rule rule, Permission permission, String description);
 
-    RolePermission createRolePermission(final Role role, final Rule rule, final Permission permission, final String description);
     /**
      * updateRolePermission updates the order/position of an role permission
      * @param role The role whose permissions needs to be re-ordered
      * @param newOrder The new list of ordered role permissions
      */
-    boolean updateRolePermission(final Role role, final List<RolePermission> newOrder);
-    boolean updateRolePermission(final Role role, final RolePermission rolePermission, final Permission permission);
-    boolean deleteRolePermission(final RolePermission rolePermission);
+    boolean updateRolePermission(Role role, List<RolePermission> newOrder);
 
+    boolean updateRolePermission(Role role, RolePermission rolePermission, Permission permission);
+
+    boolean deleteRolePermission(RolePermission rolePermission);
+
+    /**
+     *  List all roles configured in the database. Roles that have the type {@link RoleType#Admin} will not be shown for users that are not 'root admin'.
+     */
     List<Role> listRoles();
-    List<Role> findRolesByName(final String name);
-    List<Role> findRolesByType(final RoleType roleType);
-    List<RolePermission> findAllPermissionsBy(final Long roleId);
+
+    /**
+     *  Find all roles that have the giving {@link String} as part of their name.
+     *  If the user calling the method is not a 'root admin', roles of type {@link RoleType#Admin} wil lbe removed of the returned list.
+     */
+    List<Role> findRolesByName(String name);
+
+    /**
+     *  Find all roles by {@link RoleType}. If the role type is {@link RoleType#Admin}, the calling account must be a root admin, otherwise we return an empty list.
+     */
+    List<Role> findRolesByType(RoleType roleType);
+
+    List<RolePermission> findAllPermissionsBy(Long roleId);
 }

api/src/org/apache/cloudstack/acl/RoleService.java

@@ -17,31 +17,25 @@
 
 package org.apache.cloudstack.api.command.admin.acl;
 
-import com.cloud.exception.ConcurrentOperationException;
-import com.cloud.exception.InsufficientCapacityException;
-import com.cloud.exception.NetworkRuleConflictException;
-import com.cloud.exception.ResourceAllocationException;
-import com.cloud.exception.ResourceUnavailableException;
-import com.cloud.user.Account;
-import com.google.common.base.Strings;
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.List;
+
 import org.apache.cloudstack.acl.Role;
 import org.apache.cloudstack.acl.RoleType;
 import org.apache.cloudstack.api.APICommand;
 import org.apache.cloudstack.api.ApiConstants;
 import org.apache.cloudstack.api.BaseCmd;
 import org.apache.cloudstack.api.Parameter;
-import org.apache.cloudstack.api.ServerApiException;
 import org.apache.cloudstack.api.response.ListResponse;
 import org.apache.cloudstack.api.response.RoleResponse;
+import org.apache.commons.lang3.StringUtils;
 
-import java.util.ArrayList;
-import java.util.Collections;
-import java.util.List;
+import com.cloud.user.Account;
+import com.google.common.base.Strings;
 
-@APICommand(name = ListRolesCmd.APINAME, description = "Lists dynamic roles in CloudStack", responseObject = RoleResponse.class,
-        requestHasSensitiveInfo = false, responseHasSensitiveInfo = false,
-        since = "4.9.0",
-        authorized = {RoleType.Admin, RoleType.ResourceAdmin, RoleType.DomainAdmin})
+@APICommand(name = ListRolesCmd.APINAME, description = "Lists dynamic roles in CloudStack", responseObject = RoleResponse.class, requestHasSensitiveInfo = false, responseHasSensitiveInfo = false, since = "4.9.0", authorized = {
+        RoleType.Admin, RoleType.ResourceAdmin, RoleType.DomainAdmin})
 public class ListRolesCmd extends BaseCmd {
     public static final String APINAME = "listRoles";
 
@@ -112,13 +106,13 @@ private void setupResponse(final List<Role> roles) {
     }
 
     @Override
-    public void execute() throws ResourceUnavailableException, InsufficientCapacityException, ServerApiException, ConcurrentOperationException, ResourceAllocationException, NetworkRuleConflictException {
-        final List<Role> roles;
+    public void execute() {
+        List<Role> roles;
         if (getId() != null && getId() > 0L) {
             roles = Collections.singletonList(roleService.findRole(getId()));
-        } else if (!Strings.isNullOrEmpty(getName())) {
+        } else if (StringUtils.isNotBlank(getName())) {
             roles = roleService.findRolesByName(getName());
-        } else if (getRoleType() != null){
+        } else if (getRoleType() != null) {
             roles = roleService.findRolesByType(getRoleType());
         } else {
             roles = roleService.listRoles();

api/src/org/apache/cloudstack/api/command/admin/acl/ListRolesCmd.java

@@ -29,7 +29,6 @@
 import com.cloud.serializer.Param;
 import com.cloud.user.Account;
 
-@SuppressWarnings("unused")
 @EntityReference(value = Account.class)
 public class AccountResponse extends BaseResponse implements ResourceLimitAndCountResponse {
     @SerializedName(ApiConstants.ID)
@@ -222,7 +221,7 @@ public class AccountResponse extends BaseResponse implements ResourceLimitAndCou
 
     @SerializedName("secondarystoragetotal")
     @Param(description = "the total secondary storage space (in GiB) owned by account", since = "4.2.0")
-    private Long secondaryStorageTotal;
+    private float secondaryStorageTotal;
 
     @SerializedName("secondarystorageavailable")
     @Param(description = "the total secondary storage space (in GiB) available to be used for this account", since = "4.2.0")
@@ -501,7 +500,7 @@ public void setSecondaryStorageLimit(String secondaryStorageLimit) {
     }
 
     @Override
-    public void setSecondaryStorageTotal(Long secondaryStorageTotal) {
+    public void setSecondaryStorageTotal(float secondaryStorageTotal) {
         this.secondaryStorageTotal = secondaryStorageTotal;
     }
 

…stack/api/command/admin/acl/RoleCmd.java …stack/api/command/admin/acl/RoleCmd.javaapi/src/main/java/org/apache/cloudstack/api/command/admin/acl/RoleCmd.java renamed to api/src/org/apache/cloudstack/api/command/admin/acl/RoleCmd.java api/src/main/java/org/apache/cloudstack/api/command/admin/acl/RoleCmd.java renamed to api/src/org/apache/cloudstack/api/command/admin/acl/RoleCmd.java

@@ -165,7 +165,7 @@ public class DomainResponse extends BaseResponse implements ResourceLimitAndCoun
     private String secondaryStorageLimit;
 
     @SerializedName("secondarystoragetotal") @Param(description="the total secondary storage space (in GiB) owned by domain", since="4.2.0")
-    private Long secondaryStorageTotal;
+    private float secondaryStorageTotal;
 
     @SerializedName("secondarystorageavailable") @Param(description="the total secondary storage space (in GiB) available to be used for this domain", since="4.2.0")
     private String secondaryStorageAvailable;
@@ -399,7 +399,7 @@ public void setSecondaryStorageLimit(String secondaryStorageLimit) {
     }
 
     @Override
-    public void setSecondaryStorageTotal(Long secondaryStorageTotal) {
+    public void setSecondaryStorageTotal(float secondaryStorageTotal) {
         this.secondaryStorageTotal = secondaryStorageTotal;
     }
 

api/src/org/apache/cloudstack/api/response/AccountResponse.java

@@ -29,7 +29,6 @@
 import com.cloud.serializer.Param;
 
 @EntityReference(value = Project.class)
-@SuppressWarnings("unused")
 public class ProjectResponse extends BaseResponse implements ResourceLimitAndCountResponse {
 
     @SerializedName(ApiConstants.ID)
@@ -134,7 +133,7 @@ public class ProjectResponse extends BaseResponse implements ResourceLimitAndCou
 
     @SerializedName("secondarystoragetotal")
     @Param(description = "the total secondary storage space (in GiB) owned by project", since = "4.2.0")
-    private Long secondaryStorageTotal;
+    private float secondaryStorageTotal;
 
     @SerializedName("secondarystorageavailable")
     @Param(description = "the total secondary storage space (in GiB) available to be used for this project", since = "4.2.0")
@@ -414,7 +413,7 @@ public void setSecondaryStorageLimit(String secondaryStorageLimit) {
     }
 
     @Override
-    public void setSecondaryStorageTotal(Long secondaryStorageTotal) {
+    public void setSecondaryStorageTotal(float secondaryStorageTotal) {
         this.secondaryStorageTotal = secondaryStorageTotal;
     }
 

api/src/org/apache/cloudstack/api/response/DomainResponse.java

@@ -54,7 +54,7 @@ public interface ResourceLimitAndCountResponse {
 
     public void setSecondaryStorageLimit(String secondaryStorageLimit);
 
-    public void setSecondaryStorageTotal(Long secondaryStorageTotal);
+    public void setSecondaryStorageTotal(float secondaryStorageTotal);
 
     public void setSecondaryStorageAvailable(String secondaryStorageAvailable);
 

api/src/org/apache/cloudstack/api/response/ProjectResponse.java

@@ -37,6 +37,11 @@ public class IsoProcessor extends AdapterBase implements Processor {
 
     @Override
     public FormatInfo process(String templatePath, ImageFormat format, String templateName) {
+      return process(templatePath, format, templateName, 0);
+    }
+
+   @Override
+    public FormatInfo process(String templatePath, ImageFormat format, String templateName, long processTimeout) {
         if (format != null) {
             s_logger.debug("We don't handle conversion from " + format + " to ISO.");
             return null;