USHIFT-966 USHIFT-973: microshift e2e test suite by pmtk · Pull Request #1557 · openshift/microshift

pmtk · 2023-03-23T16:40:44Z

No description provided.

openshift-ci · 2023-03-23T16:41:31Z

@pmtk: GitHub didn't allow me to request PR reviews from the following users: pmtk.

Note that only openshift members and repo collaborators can review this PR, and authors cannot review their own PRs.

Details

In response to this:

/cc

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

pmtk · 2023-03-23T16:41:51Z

being tested in openshift/release#37629

dhellmann · 2023-03-23T20:44:47Z

e2e/tests/0030-router-smoke-test.sh

Can we split these test scripts up so it is possible to run them against a local deployment, too?

e2e/helpers.sh

e2e/main.sh

e2e/tests/0030-router-smoke-test.sh

e2e/tests/0050-greenboot.sh

dhellmann · 2023-04-03T19:13:51Z

e2e/cluster-debug-info.sh

At what point can this script just become a call to sos report?

As soon as the sos version we need is publicly available this script may be retired.

I don't remember the details, but I think we still don't have everything in SOS? Does it have "feature parity" with the cluster-debug-info.sh?

I take ^ back. Just looked at sos from master and it's pretty dope.
@dhellmann @pacevedom what do you think about leaving couple commands in the job log to have quickly glanceable info in case of error? Like oc get {pods,nodes} and maybe oc get events -A --sort-by=.metadata.creationTimestamp | head -n 20. Given our problems with topolvm image pulling I'd get quickly annoyed to go through downloading and extracting sos report only to find that issue there :P

e2e/main.sh

dhellmann · 2023-04-04T21:15:53Z

e2e/main.sh

Passing shell commands to ssh to run with pipes and loops and things can make quoting pretty messy. Can we process the output of the script outside of ssh? Or can we make the script being run produce just the output we want (maybe via a command line switch)?

We also, at one point, talked about just copying all of the test code into the VM and then running everything there. I can see some issues with that, but the appeal is that is simplifies the test runner logic, at least in some cases. Does it make the implementation of the tests harder, though?

I feel that tests like router/lb smokes would not be as sane ran from within.
Also - reboot test (which I haven't ported(? - there's nothing to port there actually...) yet)

I know that current state of the PR is "in between" approaches, but I didn't wanted to make too many changes to existing tests (esp greenboot) hoping for iterative approach.

Re that monstrosity after pipe - it just adds timestamp to beginning of the line, somehow I couldn't find moreutils package on RHEL (which would provide ts and that would be much simpler).
Also, it's not needed at all - just put it there because I noticed that greenboot check takes way too long (pod restart check takes 50s for each namespace sequentially)

I just remembered testing I've done simulating "power loss during initialization" - test similar to this (or just "do 100 reboots") has no place running from the microshift itself.

Having external tests runner feels cleaner to me, but there's nothing prohibiting the test suite from the microshift (like selected tests running on dev vm against dev vm), but that's probably not what you meant?
Are we talking about test runner on dev VM or just having test cases in scripts, copied to remote and executed there? Even though bash scripts are much simpler at times due to being native to shell (duh) or set -x providing full debug info, they feel like a mess.

I'm sorry, yes, you've pointed out that reboot case to me a few times before, and I keep forgetting it.

I'm trying to minimize the complexity of the commands embedded in the ssh arguments. Maybe we can isolate just those cases as temporary scripts? A guideline like "if you wouldn't type it on the command line, don't pass it to ssh as an argument" could give us a heuristic. Maybe that's stricter than needed, though?

openshift-ci-robot · 2023-04-05T07:25:04Z

@pmtk: This pull request references USHIFT-973 which is a valid jira issue.

Details

In response to this:

/cc

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

pmtk · 2023-04-06T06:44:51Z

/label tide/merge-method-squash

openshift-ci-robot · 2023-04-06T13:42:02Z

@pmtk: This pull request references USHIFT-973 which is a valid jira issue.

Details

In response to this:

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

dhellmann

I'm happy with this version. I'll leave the PR open to let some of the rest of the team look at it, since there are others who have been following the work more closely than I have.

ggiguash · 2023-04-10T04:44:08Z

e2e/main.sh

+set -euo pipefail
+
+SCRIPT_DIR="$(readlink -f "$(dirname "${BASH_SOURCE[0]}")")"
+OUTPUT_DIR="${ARTIFACT_DIR:-_output}/microshift-e2e-$(date +'%Y%m%d-%H%M%S')/"


Can we make the default _output value relative to the SCRIPT_DIR so that it's a full path?

ggiguash · 2023-04-10T04:45:15Z

e2e/main.sh

+    echo ""
+    echo " Script expects two environmental variables:"
+    echo "   - USHIFT_IP"
+    echo "   - USHIFT_USER"


Can we set 127.0.0.1 and microshift defaults for those? This should facilitate easier local execution of the scripts on the dev machine.

There will be a class of tests that cannot be run locally (anything requiring a reboot, for example. That won't be all of the tests, but for ease of development of the test suite we should be consistent with running the tests remotely.

e2e/main.sh

e2e/tests/0050-greenboot.sh

e2e/tests/0100-reboot.sh

pmtk · 2023-04-11T07:04:08Z

/hold

pmtk · 2023-04-11T11:23:57Z

/unhold

ggiguash · 2023-04-11T13:01:43Z

I'm approving this PR and putting it on hold. Please unhold when ready to merge.
/lgtm
/hold
/retest

pmtk · 2023-04-11T13:13:49Z

/test e2e-router-smoke-test

pacevedom · 2023-04-11T13:24:35Z

e2e/main.sh

+    ssh_cmd 'cat << EOF | sudo tee /etc/microshift/config.yaml
+---
+apiServer:
+  subjectAltNames:
+  - '"${USHIFT_IP}"'
+EOF' &>"${output_dir}/0001-setup.log"


Is this needed? Should it remain as part of the installation?

But it's only done in CI, missing for local VMs (unless someone does it manually). I'd prefer using IPs over hostnames if we can

Tests only require a host for the execution, be that a hostname or an IP, right? I would put the requirement on configuring the system outside of the tests so that they are generic for any environment.
In CI you have that as part of the install steps, and in a dev environment you may get away with default configuration. For example, I use "microshift-rhel9" as hostname for my VM and never use its IP, therefore the default config is ok.

pmtk · 2023-04-11T15:09:31Z

/unhold

dhellmann · 2023-04-11T19:34:12Z

/lgtm

openshift-ci · 2023-04-11T19:34:52Z

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: dhellmann, ggiguash, pmtk

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Details

Needs approval from an approver in each of these files:

~~OWNERS~~ [dhellmann,ggiguash,pmtk]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

openshift-ci · 2023-04-11T20:44:43Z

@pmtk: all tests passed!

Full PR test history. Your PR dashboard.

Details

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. I understand the commands that are listed here.

openshift-ci bot added the do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. label Mar 23, 2023

openshift-ci bot added the approved Indicates a PR has been approved by an approver from all required OWNERS files. label Mar 23, 2023

dhellmann reviewed Mar 23, 2023

View reviewed changes

pmtk force-pushed the merge-e2e-tests branch from d47b007 to 5118527 Compare March 30, 2023 09:44

dhellmann reviewed Mar 30, 2023

View reviewed changes

e2e/helpers.sh Outdated Show resolved Hide resolved

e2e/helpers.sh Outdated Show resolved Hide resolved

e2e/main.sh Outdated Show resolved Hide resolved

e2e/tests/0030-router-smoke-test.sh Outdated Show resolved Hide resolved

e2e/tests/0050-greenboot.sh Outdated Show resolved Hide resolved

dhellmann reviewed Apr 3, 2023

View reviewed changes

dhellmann reviewed Apr 4, 2023

View reviewed changes

pmtk changed the title ~~WIP microshift specific tests~~ WIP USHIFT-966 USHIFT-973: microshift e2e test suite Apr 5, 2023

openshift-ci-robot added the jira/valid-reference Indicates that this PR references a valid Jira ticket of any type. label Apr 5, 2023

pmtk force-pushed the merge-e2e-tests branch 4 times, most recently from 2023943 to c642ae8 Compare April 5, 2023 15:42

openshift-merge-robot added the needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. label Apr 5, 2023

openshift-ci bot added the tide/merge-method-squash Denotes a PR that should be squashed by tide when it merges. label Apr 6, 2023

pmtk force-pushed the merge-e2e-tests branch from 38fd063 to 0366f16 Compare April 6, 2023 13:40

openshift-merge-robot removed the needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. label Apr 6, 2023

pmtk changed the title ~~WIP USHIFT-966 USHIFT-973: microshift e2e test suite~~ USHIFT-966 USHIFT-973: microshift e2e test suite Apr 6, 2023

openshift-ci bot removed the do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. label Apr 6, 2023

dhellmann reviewed Apr 6, 2023

View reviewed changes

ggiguash suggested changes Apr 10, 2023

View reviewed changes

pacevedom mentioned this pull request Apr 10, 2023

OCPBUGS-11395: Remove stale kubeconfigs #1643

Merged

openshift-ci bot added the do-not-merge/hold Indicates that a PR should not merge because someone has issued a /hold command. label Apr 11, 2023

microshift specific tests

6d0b594

pmtk added 9 commits April 11, 2023 12:01

linter fixes

9d8efb5

remove timestamps when running greenboot check

b350f84

'typo' :)

ba64bf5

absolute path to _output

88c60bb

wrapper func for ssh

845dd35

Change error msg

e4ed822

remove unnecessary oc get

0980e32

allow for consecutive tests without reprovisioning microshift

ec645aa

extend greenboot trap EXIT

ab99122

pmtk force-pushed the merge-e2e-tests branch from a67a2be to 61e5218 Compare April 11, 2023 11:23

openshift-ci bot removed the do-not-merge/hold Indicates that a PR should not merge because someone has issued a /hold command. label Apr 11, 2023

openshift-ci bot added the do-not-merge/hold Indicates that a PR should not merge because someone has issued a /hold command. label Apr 11, 2023

openshift-ci bot assigned ggiguash Apr 11, 2023

openshift-ci bot added the lgtm Indicates that a PR is ready to be merged. label Apr 11, 2023

reword

ebc0c97

pmtk force-pushed the merge-e2e-tests branch from 61e5218 to ebc0c97 Compare April 11, 2023 13:17

openshift-ci bot removed the lgtm Indicates that a PR is ready to be merged. label Apr 11, 2023

pacevedom reviewed Apr 11, 2023

View reviewed changes

openshift-ci bot removed the do-not-merge/hold Indicates that a PR should not merge because someone has issued a /hold command. label Apr 11, 2023

openshift-ci bot assigned dhellmann Apr 11, 2023

openshift-ci bot added the lgtm Indicates that a PR is ready to be merged. label Apr 11, 2023

openshift-merge-robot merged commit 5c81392 into openshift:main Apr 11, 2023

pmtk deleted the merge-e2e-tests branch April 12, 2023 06:26

Conversation

pmtk commented Mar 23, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

openshift-ci bot commented Mar 23, 2023

Uh oh!

pmtk commented Mar 23, 2023

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

openshift-ci-robot commented Apr 5, 2023 • edited by openshift-ci bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

pmtk commented Apr 6, 2023

Uh oh!

openshift-ci-robot commented Apr 6, 2023 • edited by openshift-ci bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

dhellmann left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

ggiguash Apr 10, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

pmtk commented Apr 11, 2023

Uh oh!

pmtk commented Apr 11, 2023

Uh oh!

ggiguash commented Apr 11, 2023

Uh oh!

pmtk commented Apr 11, 2023

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

pmtk commented Apr 11, 2023

Uh oh!

dhellmann commented Apr 11, 2023

Uh oh!

openshift-ci bot commented Apr 11, 2023

Uh oh!

openshift-ci bot commented Apr 11, 2023

Uh oh!

pmtk commented Mar 23, 2023 •

edited

Loading

openshift-ci-robot commented Apr 5, 2023 •

edited by openshift-ci bot

Loading

openshift-ci-robot commented Apr 6, 2023 •

edited by openshift-ci bot

Loading

ggiguash Apr 10, 2023 •

edited

Loading