Skip to content

USHIFT-6586: introduce tls-scanner test to optional suite #6266

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
eslutsky wants to merge 1 commit into
base: main
Choose a base branch
from
+153 −0
Open

USHIFT-6586: introduce tls-scanner test to optional suite #6266

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
40 changes: 40 additions & 0 deletions test/assets/tls-scanner/cluster-reader-clusterrole.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,40 @@
# ClusterRole equivalent to OpenShift's cluster-reader for MicroShift.
# MicroShift does not ship this role; tls-scanner deploy.sh expects it.
# Read-only (get, list, watch) on core and common resources.
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
name: cluster-reader
rules:
- apiGroups: [""]
resources:
- configmaps
- endpoints
- namespaces
- nodes
- persistentvolumeclaims
- pods
- podtemplates
- replicationcontrollers
- resourcequotas
- services
verbs: [get, list, watch]
- apiGroups: ["apps"]
resources:
- daemonsets
- deployments
- replicasets
- statefulsets
verbs: [get, list, watch]
- apiGroups: ["batch"]
resources:
- cronjobs
- jobs
verbs: [get, list, watch]
- apiGroups: ["rbac.authorization.k8s.io"]
resources:
- clusterrolebindings
- clusterroles
- rolebindings
- roles
verbs: [get, list, watch]
113 changes: 113 additions & 0 deletions test/suites/optional/tls-scanner.robot
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,113 @@
*** Settings ***
Documentation Test tls-scanner tool with MicroShift host-based scanning.
... Clones openshift/tls-scanner, deploys the scanner job with
... scanner-job-microshift.yaml.template and SCAN_MODE=host,
... waits for completion, and collects results.
... See: https://github.com/openshift/tls-scanner

Library OperatingSystem
Library Process
Library String
Resource ../../resources/common.resource
Resource ../../resources/kubeconfig.resource
Resource ../../resources/oc.resource

Suite Setup Setup Suite With Namespace
Suite Teardown Teardown Suite With Namespace

Test Tags tls-scanner security optional


*** Variables ***
# Set by Suite Setup (common.resource / kubeconfig.resource):
${NAMESPACE} default
${KUBECONFIG} ${EMPTY}
# External: full tag of the scanner image to use (e.g. quay.io/my-org/tls-scanner:latest)
${SCANNER_IMAGE} registry.ci.openshift.org/ocp/4.22:tls-scanner-tool

# TODO: change eslutsky to openshift once https://github.com/openshift/tls-scanner/pull/17 merged
${TLS_SCANNER_REPO} https://github.com/eslutsky/tls-scanner


${TLS_SCANNER_DIR} ${EMPTY}
${TLS_SCANNER_JOB_NAME} tls-scanner-job
${JOB_WAIT_TIMEOUT} 10min
${CLUSTER_READER_MANIFEST} ./assets/tls-scanner/cluster-reader-clusterrole.yaml


*** Test Cases ***
TLS Scanner Host Scan Completes And Produces Artifacts
[Documentation] Clone tls-scanner, verify scanner image is available,
... deploy the scan job in host mode for MicroShift, wait for completion,
... and collect results (results.json, results.csv, scan.log).
[Setup] Run Keywords
... Check Required Scanner Variables
... Clone TLS Scanner Repo
... Ensure Cluster Reader Role Exists
Deploy TLS Scanner Job
Copy Scan Results Artifacts

[Teardown] Run Keywords
... Cleanup TLS Scanner Job


*** Keywords ***
Check Required Scanner Variables
[Documentation] Fail if SCANNER_IMAGE is not set.
Should Not Be Empty ${SCANNER_IMAGE}
... SCANNER_IMAGE must be set (full image tag, e.g. quay.io/my-org/tls-scanner:latest)

Ensure Cluster Reader Role Exists
[Documentation] Create cluster-reader ClusterRole for MicroShift (not shipped by default).
... deploy.sh expects this OpenShift role to exist for the scanner ServiceAccount.
Oc Apply -f ${CLUSTER_READER_MANIFEST}
Comment on lines +60 to +63
Copy link

@coderabbitai coderabbitai bot Feb 24, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

⚠️ Potential issue | 🟡 Minor

cluster-reader ClusterRole is never deleted after the test.

deploy.sh cleanup only tears down what deploy.sh deploy created. The separately applied ClusterRole persists in the cluster, and Teardown Suite With Namespace won't touch cluster-scoped resources. Add an explicit deletion in teardown.

🧹 Proposed fix — add to `Cleanup TLS Scanner Job` 
 Cleanup TLS Scanner Job
     [Documentation]    Remove the scanner job and RBAC via deploy.sh cleanup.
     ${result}=    Run Keyword And Ignore Error    Process.Run Process    ./deploy.sh cleanup
     ...    cwd=${TLS_SCANNER_DIR}
     ...    env:KUBECONFIG=${KUBECONFIG}
     ...    env:NAMESPACE=${NAMESPACE}
     ...    shell=True    timeout=60s
     IF    "${result[0]}" == "PASS"    Log    TLS scanner job cleanup completed
+    IF    "${result[0]}" == "FAIL"    Log    TLS scanner cleanup failed (non-fatal): ${result[1]}    level=WARN
+    Run Keyword And Ignore Error    Oc Delete    -f ${CLUSTER_READER_MANIFEST} --ignore-not-found
     IF    '${TLS_SCANNER_DIR}' != ''
         Run Keyword And Ignore Error    Remove Directory    ${TLS_SCANNER_DIR}    recursive=True
     END
🤖 Prompt for AI Agents 
Verify each finding against the current code and only fix it if needed.

In `@test/suites/optional/tls-scanner.robot` around lines 56 - 59, The
cluster-reader ClusterRole applied in the test is never removed; update the
teardown keyword "Cleanup TLS Scanner Job" to explicitly delete that
cluster-scoped resource by invoking Oc Delete on ${CLUSTER_READER_MANIFEST}
(e.g., add an "Oc Delete    -f ${CLUSTER_READER_MANIFEST}" step inside the
Cleanup TLS Scanner Job teardown) so the cluster-reader ClusterRole is removed
after the suite.


Clone TLS Scanner Repo
[Documentation] Clone openshift/tls-scanner into a temporary directory.
${rand}= Generate Random String 8 [LOWER]
VAR ${workdir}= /tmp/tls-scanner-${rand}
Create Directory ${workdir}
VAR ${TLS_SCANNER_DIR}= ${workdir} scope=SUITE
${result}= Process.Run Process git clone --depth 1 ${TLS_SCANNER_REPO} .
... cwd=${TLS_SCANNER_DIR} shell=True timeout=120s
Log ${result.stdout}
Log ${result.stderr}
Should Be Equal As Integers ${result.rc} 0 msg=Failed to clone tls-scanner repo

Deploy TLS Scanner Job
[Documentation] Deploy the scanner job using MicroShift host template and SCAN_MODE=host.
${result}= Process.Run Process bash -c 'bash -x ./deploy.sh deploy 2>&1'
... cwd=${TLS_SCANNER_DIR}
... env:KUBECONFIG=${KUBECONFIG}
... env:SCANNER_IMAGE=${SCANNER_IMAGE}
... env:NAMESPACE=${NAMESPACE}
... env:JOB_TEMPLATE_FILE=scanner-job-microshift.yaml.template
... env:SCAN_MODE=host
... env:OUTPUTDIR=${OUTPUTDIR}
... shell=True timeout=${JOB_WAIT_TIMEOUT} stdout=${OUTPUTDIR}/tls-scanner-std.log
Log ${result.stdout}
Log ${result.stderr}
Should Be Equal As Integers ${result.rc} 0 msg=Failed to deploy tls-scanner job

Copy Scan Results Artifacts
[Documentation] Copy content of ${TLS_SCANNER_DIR}/artifacts to ${OUTPUTDIR}/tls-scanner-artifacts.
VAR ${dest}= ${OUTPUTDIR}/tls-scanner-artifacts
Create Directory ${dest}
OperatingSystem.Directory Should Exist ${TLS_SCANNER_DIR}/artifacts
${files}= OperatingSystem.List Files In Directory ${TLS_SCANNER_DIR}/artifacts
${count}= Get Length ${files}
Should Be True ${count} > 0 msg=No artifacts produced by tls-scanner
FOR ${f} IN @{files}
Copy File ${TLS_SCANNER_DIR}/artifacts/${f} ${dest}/
END
Log Copied scan results to ${dest}/

Cleanup TLS Scanner Job
[Documentation] Remove the scanner job and RBAC via deploy.sh cleanup.
${result}= Run Keyword And Ignore Error Process.Run Process ./deploy.sh cleanup
... cwd=${TLS_SCANNER_DIR}
... env:KUBECONFIG=${KUBECONFIG}
... env:NAMESPACE=${NAMESPACE}
... shell=True timeout=60s
IF "${result[0]}" == "PASS" Log TLS scanner job cleanup completed
Remove Directory ${TLS_SCANNER_DIR} recursive=True
Comment on lines +105 to +113
Copy link

@coderabbitai coderabbitai bot Feb 23, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

⚠️ Potential issue | 🟡 Minor

Guard teardown against empty ${TLS_SCANNER_DIR}.

If setup fails before setting the suite variable, Remove Directory can error out during teardown. Make it best‑effort and conditional.

🔧 Proposed update 
-    Remove Directory    ${TLS_SCANNER_DIR}    recursive=True
+    IF    '${TLS_SCANNER_DIR}' != ''
+        Run Keyword And Ignore Error    Remove Directory    ${TLS_SCANNER_DIR}    recursive=True
+    END
🤖 Prompt for AI Agents 
Verify each finding against the current code and only fix it if needed.

In `@test/suites/optional/tls-scanner.robot` around lines 101 - 109, The teardown
currently always calls Remove Directory on ${TLS_SCANNER_DIR}, which will error
if the variable is unset/empty; update the "Cleanup TLS Scanner Job" section to
guard the removal by checking the variable first (e.g., use IF/Run Keyword If to
test that ${TLS_SCANNER_DIR} is defined/non-empty and not just whitespace) and
only call Remove Directory when the check passes, keeping the existing Run
Keyword And Ignore Error call for deploy.sh and preserving recursive=True
behavior when removing the directory.