good information about failing to add xcp-ng 7.6.0 host missing

[alexandru-bagu]

Setting up a test environment from master I had issues adding an xcp-ng host failing without any information. Turns out

cloudstack/plugins/hypervisors/xenserver/src/main/java/com/cloud/hypervisor/xenserver/discoverer/XcpServerDiscoverer.java

Line 372 in fb1e903

s_logger.debug("other exceptions: " + e.toString(), e);

is set to output to debug any unknown errors.

The stacktrace:

ERROR [c.c.h.x.d.XcpServerDiscoverer] (qtp186780379-49:ctx-a9ff61f5 ctx-a4d8e914) (logid:2bc049f8) other exceptions: com.cloud.utils.exception.CloudRuntimeException: Unable to create master connection to host(192.168.100.11) , due to org.apache.xmlrpc.XmlRpcException: Failed to read server's response: Certificates do not conform to algorithm constraints
com.cloud.utils.exception.CloudRuntimeException: Unable to create master connection to host(192.168.100.11) , due to org.apache.xmlrpc.XmlRpcException: Failed to read server's response: Certificates do not conform to algorithm constraints
at com.cloud.hypervisor.xenserver.resource.XenServerConnectionPool.getConnect(XenServerConnectionPool.java:168)
at com.cloud.hypervisor.xenserver.discoverer.XcpServerDiscoverer.find(XcpServerDiscoverer.java:215)
at com.cloud.resource.ResourceManagerImpl.discoverHostsFull(ResourceManagerImpl.java:767)
...

Notice it says "ERROR" at the begining. That is because I changed it to log.error so I could actually see the error. The obvious solution would be to enable DEBUG when something fails but considering this operation is seldom done why even use debug for logging such errors. By default the DEBUG flag is not set and setting it is not necessarily trivial.

Without any debug flags on the stacktrace looks like this:

WARN [c.c.r.ResourceManagerImpl] (qtp561469384-298:ctx-39d175c2 ctx-6647769a) (logid:d6ca3eee) Unable to find the server resources at http://192.168.100.11
INFO [c.c.u.e.CSExceptionErrorCode] (qtp561469384-298:ctx-39d175c2 ctx-6647769a) (logid:d6ca3eee) Could not find exception: com.cloud.exception.DiscoveryException in error code list for exceptions
WARN [o.a.c.a.c.a.h.AddHostCmd] (qtp561469384-298:ctx-39d175c2 ctx-6647769a) (logid:d6ca3eee) Exception:
com.cloud.exception.DiscoveryException: Unable to add the host
at com.cloud.resource.ResourceManagerImpl.discoverHostsFull(ResourceManagerImpl.java:826)
...

"Caused by: java.security.cert.CertPathValidatorException: Algorithm constraints check failed on keysize limits. RSA 1024bit key used with certificate: CN=192.168.100.11. Usage was tls server"

Oddly enough on Windows with AdoptOpenJDK build I did not get any errors, so I assume some jdk builds have different constraints for certificates.
Also running mvn jetty:run -X does not actually show the debug stacktrace, but it does show tons of other debug lines. Odd.

The solution is to regenerate the xcp-ng certificate after the key size is changed. Steps to do on xcp-ng server:

1. modify /opt/xensource/libexec/generate_ssl_cert; find line with "openssl genrsa" and change the 3rd parameter to be anything more than (or equal to) 1024. 2048 or 4096 is better.
2. move the previous certificate and keep it as a backup: mv /etc/xensource/xapi-ssl.pem /etc/xensource/xapi-ssl.pem.bak
3. regenerate certificate: /opt/xensource/libexec/generate_ssl_cert /etc/xensource/xapi-ssl.pem $(hostname -f) && service xapi restart

[rohityadavcloud]

@alexandru-bagu thanks for sharing. Assuming CloudStack doesnot generate those certificates, this sounds like a XCP-ng issue or something we can document in our docs (in the add host section): https://github.com/apache/cloudstack-documentation

[alexandru-bagu]

I agree that the certificate issue itself is not CloudStack's problem. The issue however is that unless you have debug flag set you will not know why the host could not be added. The change I am suggesting is related to logging of the error, nothing related to UI or behavior. Because the operation of adding a host is seldom done and it's only by user request, all information for failure should be logged properly. I don't believe logging critical information with log.debug is correct, even if the exception is unknown to CloudStack.

Had CloudStack told me what the issue was (using default log settings, without any tinkering on my part) I would have been able to figure out a solution much faster.

[DaanHoogland]

@alexandru-bagu , I took the liberty to rename your issue so it describes the issue with-/from cloudstack point of view.
log levels are a long going issue with cloudstack and this is a good example of it. The argument people often use is that cloudstack is programmed to run in production with DEBUG on. bit I think we should improve this. Do you have a PR planned for this?

[alexandru-bagu]

It's just a matter of modifying the line I highlighted in the OP from debug to error. I will submit a pull request soon.

[weizhouapache]

a workaround could be: change RSA keySize < 2048 to RSA keySize < 1024 in /etc/crypto-policies/back-ends/java.config

# cat /etc/crypto-policies/back-ends/java.config
jdk.tls.ephemeralDHKeySize=2048
jdk.certpath.disabledAlgorithms=MD2, MD5, DSA, RSA keySize < 2048
jdk.tls.disabledAlgorithms=DH keySize < 2048, SSLv2, SSLv3, TLSv1, TLSv1.1, DHE_DSS, RSA_EXPORT, DHE_DSS_EXPORT, DHE_RSA_EXPORT, DH_DSS_EXPORT, DH_RSA_EXPORT, DH_anon, ECDH_anon, DH_RSA, DH_DSS, ECDH, 3DES_EDE_CBC, DES_CBC, RC4_40, RC4_128, DES40_CBC, RC2, HmacMD5
jdk.tls.legacyAlgorithms=