Retry redfish requests

[GabrielBrascher]

Description

It is not common, but HTTP requests can fail due to connection issues. In order to mitigate such situations and also improve logging, this PR enhances the Redfish request handling by adding an execution flow for re-trying HTTP requests; the retry happens only if the global settings redfish.retries is set to 1 or more retries; default is of 2 (two). One can disable the retries by setting redfish.retries to 0 (zero).

Types of changes

• Breaking change (fix or feature that would cause existing functionality to change)
• New feature (non-breaking change which adds functionality)
• Bug fix (non-breaking change which fixes an issue)
• Enhancement (improves an existing feature and functionality)
• Cleanup (Code refactoring and cleanup, that may add test cases)

How Has This Been Tested?

Configured a test environment to ignore SSL while using HTTPS for Redfish requests. In such cases, IOExcetions will happen on a higher frequency than any other setup due to the exception javax.net.ssl.SSLHandshakeException: Remote host terminated the handshake.

The following log shows GET requests that failed a couple of times, the successful request happened at the third attempt:

Failed to execute HTTP GET request [URL: https://oob-redfish.net/redfish/v1/Systems/]. Executing the request again.
Retry HTTP GET request [URL: https://oob-redfish.net/redfish/v1/Systems/], attempt 1/2.
Failed to execute HTTP GET request retry attempt 1/2 [URL: https://oob-redfish.net/redfish/v1/Systems/] due to exception javax.net.ssl.SSLHandshakeException: Remote host terminated the handshake
Retry HTTP GET request [URL: https://oob-redfish.com/redfish/v1/Systems/], attempt 2/2.
Successfully executed HTTP GET request [URL: https://oob-redfish.net/redfish/v1/Systems/].
Retrieved System ID 'System.Embedded.1' with request 'GET: https://oob-redfish.com/redfish/v1/Systems/'
Retrieved System power state 'On' with request 'GET: https://oob-redfish.com/redfish/v1/Systems/System.Embedded.1'

Note: Ignoring SSL is not ideal but the redfish out-of-band driver works fine in such case (we have been using it for a couple of months already). However, SSLHandshakeException (grandchild class of IOException) exceptions can be thrown. Nonetheless, ADMINS should have the freedom to still use Refish requests in such cases therefore it was and it has been working OK.

[GabrielBrascher]

@blueoragutan package

[GabrielBrascher]

@blueorangutan package

[blueorangutan]

@GabrielBrascher a Jenkins job has been kicked to build packages. I'll keep you posted as I make progress.

[blueorangutan]

Packaging result: ✔centos7 ✔centos8 ✔debian. JID-2093

[DaanHoogland]

code looks ok, unit tests and integration test directives supplied.

[rohityadavcloud]

Merging based on Travis tests, changes are strictly in redfish oobm plugin that can't be tested with kvm/smoketest.