OCPBUGS-8425: Fix the kustomization procedure not to exit with fatal error by ggiguash · Pull Request #1463 · openshift/microshift

ggiguash · 2023-03-08T06:19:32Z

Closes OCPBUGS-8425

Example of a failed run, which backs off after 1m of retries, logs an error and proceeds.

kustomizer I0312 18:30:20.289488  180399 manager.go:114] Starting kustomizer
kustomizer I0312 18:30:20.289524  180399 apply.go:64] No kustomization found at /usr/lib/microshift/manifests/kustomization.yaml
kustomizer I0312 18:30:20.289532  180399 apply.go:57] Applying kustomization at /etc/microshift/manifests/kustomization.yaml 
kustomizer I0312 18:30:20.738705  180399 apply.go:71] Applying kustomization failed: accumulating resources: accumulating resources from 'busybox.yaml': MalformedYAMLError: yaml: line 20: could not find expected ':' in File: busybox.yaml. Retrying in 10s.
kustomizer I0312 18:30:30.790331  180399 apply.go:71] Applying kustomization failed: accumulating resources: accumulating resources from 'busybox.yaml': MalformedYAMLError: yaml: line 20: could not find expected ':' in File: busybox.yaml. Retrying in 10s.
kustomizer I0312 18:30:40.806277  180399 apply.go:71] Applying kustomization failed: accumulating resources: accumulating resources from 'busybox.yaml': MalformedYAMLError: yaml: line 20: could not find expected ':' in File: busybox.yaml. Retrying in 10s.
kustomizer I0312 18:30:50.811309  180399 apply.go:71] Applying kustomization failed: accumulating resources: accumulating resources from 'busybox.yaml': MalformedYAMLError: yaml: line 20: could not find expected ':' in File: busybox.yaml. Retrying in 10s.
kustomizer I0312 18:31:00.794630  180399 apply.go:71] Applying kustomization failed: accumulating resources: accumulating resources from 'busybox.yaml': MalformedYAMLError: yaml: line 20: could not find expected ':' in File: busybox.yaml. Retrying in 10s.
kustomizer I0312 18:31:10.800532  180399 apply.go:71] Applying kustomization failed: accumulating resources: accumulating resources from 'busybox.yaml': MalformedYAMLError: yaml: line 20: could not find expected ':' in File: busybox.yaml. Retrying in 10s.
kustomizer I0312 18:31:20.785916  180399 apply.go:71] Applying kustomization failed: accumulating resources: accumulating resources from 'busybox.yaml': MalformedYAMLError: yaml: line 20: could not find expected ':' in File: busybox.yaml. Retrying in 10s.
kustomizer I0312 18:31:20.857375  180399 apply.go:71] Applying kustomization failed: accumulating resources: accumulating resources from 'busybox.yaml': MalformedYAMLError: yaml: line 20: could not find expected ':' in File: busybox.yaml. Retrying in 10s.
kustomizer E0312 18:31:20.857418  180399 apply.go:59] Applying kustomization at /etc/microshift/manifests/kustomization.yaml failed: timed out waiting for the condition. Giving up.
kustomizer I0312 18:31:20.857431  180399 manager.go:119] kustomizer completed

openshift-ci-robot · 2023-03-08T06:19:36Z

@ggiguash: This pull request references Jira Issue OCPBUGS-8425, which is invalid:

expected the bug to target the "4.14.0" version, but it targets "4.14" instead

Comment /jira refresh to re-evaluate validity if changes to the Jira bug are made, or edit the title of this pull request to link to a different bug.

The bug has been updated to refer to the pull request using the external bug tracker.

Details

In response to this:

Closes OCPBUGS-8425

Example of a failed run, which backs off after 3 retries, logs an error and proceeds.

kustomizer I0308 06:14:12.668042  239986 manager.go:114] Starting kustomizer
kustomizer I0308 06:14:12.668088  239986 apply.go:66] No kustomization found at /usr/lib/microshift/manifests/kustomization.yaml
kustomizer I0308 06:14:12.668307  239986 apply.go:59] Applying kustomization at /etc/microshift/manifests/kustomization.yaml 
kustomizer I0308 06:14:13.042466  239986 apply.go:81] Applying kustomization failed: accumulating resources: accumulating resources from 'busybox.yaml': MalformedYAMLError: yaml: line 20: could not find expected ':' in File: busybox.yaml. Retrying in 10s.
kustomizer I0308 06:14:23.117034  239986 apply.go:81] Applying kustomization failed: accumulating resources: accumulating resources from 'busybox.yaml': MalformedYAMLError: yaml: line 20: could not find expected ':' in File: busybox.yaml. Retrying in 10s.
kustomizer I0308 06:14:33.102860  239986 apply.go:81] Applying kustomization failed: accumulating resources: accumulating resources from 'busybox.yaml': MalformedYAMLError: yaml: line 20: could not find expected ':' in File: busybox.yaml. Retrying in 10s.
kustomizer I0308 06:14:43.042649  239986 apply.go:77] Applying kustomization failed: reached maximum of 3 retries
kustomizer E0308 06:14:43.042695  239986 apply.go:61] Applying kustomization at /etc/microshift/manifests/kustomization.yaml failed: reached maximum of 3 retries. Giving up.
kustomizer I0308 06:14:43.042710  239986 manager.go:119] kustomizer completed

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

ggiguash · 2023-03-08T06:20:30Z

/jira refresh

openshift-ci-robot · 2023-03-08T06:20:39Z

@ggiguash: This pull request references Jira Issue OCPBUGS-8425, which is valid. The bug has been moved to the POST state.

3 validation(s) were run on this bug

bug is open, matching expected state (open)
bug target version (4.14.0) matches configured target version for branch (4.14.0)
bug is in the state ASSIGNED, which is one of the valid states (NEW, ASSIGNED, POST)

Requesting review from QA contact:
/cc @jogeo

Details

In response to this:

/jira refresh

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

pkg/kustomize/apply.go

openshift-ci-robot · 2023-03-08T14:05:56Z

@ggiguash: This pull request references Jira Issue OCPBUGS-8425, which is valid.

3 validation(s) were run on this bug

bug is open, matching expected state (open)
bug target version (4.14.0) matches configured target version for branch (4.14.0)
bug is in the state POST, which is one of the valid states (NEW, ASSIGNED, POST)

Requesting review from QA contact:
/cc @jogeo

Details

In response to this:

Closes OCPBUGS-8425

Example of a failed run, which backs off after 3 retries, logs an error and proceeds.

kustomizer I0308 14:04:46.830239    9777 manager.go:114] Starting kustomizer
kustomizer I0308 14:04:46.830257    9777 apply.go:66] No kustomization found at /usr/lib/microshift/manifests/kustomization.yaml
kustomizer I0308 14:04:46.830754    9777 apply.go:59] Applying kustomization at /etc/microshift/manifests/kustomization.yaml 
kustomizer I0308 14:04:47.302910    9777 apply.go:79] Applying kustomization failed: accumulating resources: accumulating resources from 'busybox.yaml': MalformedYAMLError: yaml: line 20: could not find expected ':' in File: busybox.yaml. Retrying in 10s.
kustomizer I0308 14:04:57.354895    9777 apply.go:79] Applying kustomization failed: accumulating resources: accumulating resources from 'busybox.yaml': MalformedYAMLError: yaml: line 20: could not find expected ':' in File: busybox.yaml. Retrying in 10s.
kustomizer I0308 14:05:07.349669    9777 apply.go:79] Applying kustomization failed: accumulating resources: accumulating resources from 'busybox.yaml': MalformedYAMLError: yaml: line 20: could not find expected ':' in File: busybox.yaml. Retrying in 10s.
kustomizer E0308 14:05:17.303639    9777 apply.go:61] Applying kustomization at /etc/microshift/manifests/kustomization.yaml failed: reached maximum of 3 retries. Giving up.
kustomizer I0308 14:05:17.303670    9777 manager.go:119] kustomizer completed

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

pkg/kustomize/apply.go

openshift-ci-robot · 2023-03-12T18:32:33Z

@ggiguash: This pull request references Jira Issue OCPBUGS-8425, which is valid.

3 validation(s) were run on this bug

bug is open, matching expected state (open)
bug target version (4.14.0) matches configured target version for branch (4.14.0)
bug is in the state POST, which is one of the valid states (NEW, ASSIGNED, POST)

Requesting review from QA contact:
/cc @jogeo

Details

In response to this:

Closes OCPBUGS-8425

Example of a failed run, which backs off after 1m of retries, logs an error and proceeds.

kustomizer I0312 18:30:20.289488  180399 manager.go:114] Starting kustomizer
kustomizer I0312 18:30:20.289524  180399 apply.go:64] No kustomization found at /usr/lib/microshift/manifests/kustomization.yaml
kustomizer I0312 18:30:20.289532  180399 apply.go:57] Applying kustomization at /etc/microshift/manifests/kustomization.yaml 
kustomizer I0312 18:30:20.738705  180399 apply.go:71] Applying kustomization failed: accumulating resources: accumulating resources from 'busybox.yaml': MalformedYAMLError: yaml: line 20: could not find expected ':' in File: busybox.yaml. Retrying in 10s.
kustomizer I0312 18:30:30.790331  180399 apply.go:71] Applying kustomization failed: accumulating resources: accumulating resources from 'busybox.yaml': MalformedYAMLError: yaml: line 20: could not find expected ':' in File: busybox.yaml. Retrying in 10s.
kustomizer I0312 18:30:40.806277  180399 apply.go:71] Applying kustomization failed: accumulating resources: accumulating resources from 'busybox.yaml': MalformedYAMLError: yaml: line 20: could not find expected ':' in File: busybox.yaml. Retrying in 10s.
kustomizer I0312 18:30:50.811309  180399 apply.go:71] Applying kustomization failed: accumulating resources: accumulating resources from 'busybox.yaml': MalformedYAMLError: yaml: line 20: could not find expected ':' in File: busybox.yaml. Retrying in 10s.
kustomizer I0312 18:31:00.794630  180399 apply.go:71] Applying kustomization failed: accumulating resources: accumulating resources from 'busybox.yaml': MalformedYAMLError: yaml: line 20: could not find expected ':' in File: busybox.yaml. Retrying in 10s.
kustomizer I0312 18:31:10.800532  180399 apply.go:71] Applying kustomization failed: accumulating resources: accumulating resources from 'busybox.yaml': MalformedYAMLError: yaml: line 20: could not find expected ':' in File: busybox.yaml. Retrying in 10s.
kustomizer I0312 18:31:20.785916  180399 apply.go:71] Applying kustomization failed: accumulating resources: accumulating resources from 'busybox.yaml': MalformedYAMLError: yaml: line 20: could not find expected ':' in File: busybox.yaml. Retrying in 10s.
kustomizer I0312 18:31:20.857375  180399 apply.go:71] Applying kustomization failed: accumulating resources: accumulating resources from 'busybox.yaml': MalformedYAMLError: yaml: line 20: could not find expected ':' in File: busybox.yaml. Retrying in 10s.
kustomizer E0312 18:31:20.857418  180399 apply.go:59] Applying kustomization at /etc/microshift/manifests/kustomization.yaml failed: timed out waiting for the condition. Giving up.
kustomizer I0312 18:31:20.857431  180399 manager.go:119] kustomizer completed

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

dhellmann

/lgtm

openshift-ci · 2023-03-12T19:14:52Z

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: dhellmann, ggiguash, pacevedom

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Details

Needs approval from an approver in each of these files:

~~OWNERS~~ [dhellmann,ggiguash,pacevedom]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

openshift-ci · 2023-03-12T19:28:15Z

@ggiguash: all tests passed!

Full PR test history. Your PR dashboard.

Details

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. I understand the commands that are listed here.

openshift-ci-robot · 2023-03-12T19:32:59Z

@ggiguash: Jira Issue OCPBUGS-8425: All pull requests linked via external trackers have merged:

openshift/microshift#1463

Jira Issue OCPBUGS-8425 has been moved to the MODIFIED state.

Details

In response to this:

Closes OCPBUGS-8425

Example of a failed run, which backs off after 1m of retries, logs an error and proceeds.

kustomizer I0312 18:30:20.289488  180399 manager.go:114] Starting kustomizer
kustomizer I0312 18:30:20.289524  180399 apply.go:64] No kustomization found at /usr/lib/microshift/manifests/kustomization.yaml
kustomizer I0312 18:30:20.289532  180399 apply.go:57] Applying kustomization at /etc/microshift/manifests/kustomization.yaml 
kustomizer I0312 18:30:20.738705  180399 apply.go:71] Applying kustomization failed: accumulating resources: accumulating resources from 'busybox.yaml': MalformedYAMLError: yaml: line 20: could not find expected ':' in File: busybox.yaml. Retrying in 10s.
kustomizer I0312 18:30:30.790331  180399 apply.go:71] Applying kustomization failed: accumulating resources: accumulating resources from 'busybox.yaml': MalformedYAMLError: yaml: line 20: could not find expected ':' in File: busybox.yaml. Retrying in 10s.
kustomizer I0312 18:30:40.806277  180399 apply.go:71] Applying kustomization failed: accumulating resources: accumulating resources from 'busybox.yaml': MalformedYAMLError: yaml: line 20: could not find expected ':' in File: busybox.yaml. Retrying in 10s.
kustomizer I0312 18:30:50.811309  180399 apply.go:71] Applying kustomization failed: accumulating resources: accumulating resources from 'busybox.yaml': MalformedYAMLError: yaml: line 20: could not find expected ':' in File: busybox.yaml. Retrying in 10s.
kustomizer I0312 18:31:00.794630  180399 apply.go:71] Applying kustomization failed: accumulating resources: accumulating resources from 'busybox.yaml': MalformedYAMLError: yaml: line 20: could not find expected ':' in File: busybox.yaml. Retrying in 10s.
kustomizer I0312 18:31:10.800532  180399 apply.go:71] Applying kustomization failed: accumulating resources: accumulating resources from 'busybox.yaml': MalformedYAMLError: yaml: line 20: could not find expected ':' in File: busybox.yaml. Retrying in 10s.
kustomizer I0312 18:31:20.785916  180399 apply.go:71] Applying kustomization failed: accumulating resources: accumulating resources from 'busybox.yaml': MalformedYAMLError: yaml: line 20: could not find expected ':' in File: busybox.yaml. Retrying in 10s.
kustomizer I0312 18:31:20.857375  180399 apply.go:71] Applying kustomization failed: accumulating resources: accumulating resources from 'busybox.yaml': MalformedYAMLError: yaml: line 20: could not find expected ':' in File: busybox.yaml. Retrying in 10s.
kustomizer E0312 18:31:20.857418  180399 apply.go:59] Applying kustomization at /etc/microshift/manifests/kustomization.yaml failed: timed out waiting for the condition. Giving up.
kustomizer I0312 18:31:20.857431  180399 manager.go:119] kustomizer completed

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

dhellmann · 2023-03-12T22:16:05Z

/cherry-pick release-4.13

openshift-cherrypick-robot · 2023-03-12T22:16:52Z

@dhellmann: new pull request created: #1490

Details

In response to this:

/cherry-pick release-4.13

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

openshift-cherrypick-robot · 2023-03-13T04:48:45Z

@ggiguash: new pull request could not be created: failed to create pull request against openshift/microshift#release-4.13 from head openshift-cherrypick-robot:cherry-pick-1463-to-release-4.13: status code 422 not one of [201], body: {"message":"Validation Failed","errors":[{"resource":"PullRequest","code":"custom","message":"No commits between openshift:release-4.13 and openshift-cherrypick-robot:cherry-pick-1463-to-release-4.13"}],"documentation_url":"https://docs.github.com/rest/reference/pulls#create-a-pull-request"}

Details

In response to this:

/cherrypick release-4.13

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

ggiguash · 2023-03-13T04:49:01Z

/cherrypick release-4.13

openshift-cherrypick-robot · 2023-03-13T04:49:42Z

@ggiguash: new pull request could not be created: failed to create pull request against openshift/microshift#release-4.13 from head openshift-cherrypick-robot:cherry-pick-1463-to-release-4.13: status code 422 not one of [201], body: {"message":"Validation Failed","errors":[{"resource":"PullRequest","code":"custom","message":"No commits between openshift:release-4.13 and openshift-cherrypick-robot:cherry-pick-1463-to-release-4.13"}],"documentation_url":"https://docs.github.com/rest/reference/pulls#create-a-pull-request"}

Details

In response to this:

/cherrypick release-4.13

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

openshift-ci-robot added the jira/valid-reference Indicates that this PR references a valid Jira ticket of any type. label Mar 8, 2023

openshift-ci-robot added the jira/invalid-bug Indicates that a referenced Jira bug is invalid for the branch this PR is targeting. label Mar 8, 2023

openshift-ci-robot added jira/valid-bug Indicates that a referenced Jira bug is valid for the branch this PR is targeting. bugzilla/valid-bug Indicates that a referenced Bugzilla bug is valid for the branch this PR is targeting. labels Mar 8, 2023

openshift-ci-robot removed the jira/invalid-bug Indicates that a referenced Jira bug is invalid for the branch this PR is targeting. label Mar 8, 2023

openshift-ci bot requested review from jogeo, oglok and pacevedom March 8, 2023 06:20

openshift-ci bot added the approved Indicates a PR has been approved by an approver from all required OWNERS files. label Mar 8, 2023

pacevedom reviewed Mar 8, 2023

View reviewed changes

pkg/kustomize/apply.go Outdated Show resolved Hide resolved

ggiguash force-pushed the manifest_retry branch from 65ad049 to 12cb496 Compare March 8, 2023 13:15

pacevedom approved these changes Mar 8, 2023

View reviewed changes

pkg/kustomize/apply.go Outdated Show resolved Hide resolved

ggiguash force-pushed the manifest_retry branch from 12cb496 to c4e0fb9 Compare March 8, 2023 13:48

dhellmann reviewed Mar 8, 2023

View reviewed changes

pkg/kustomize/apply.go Outdated Show resolved Hide resolved

dhellmann reviewed Mar 12, 2023

View reviewed changes

pkg/kustomize/apply.go Outdated Show resolved Hide resolved

Fix the kustomization procedure not to exit with fatal error

71de253

ggiguash force-pushed the manifest_retry branch from c4e0fb9 to 71de253 Compare March 12, 2023 18:33

ggiguash changed the title ~~OCPBUGS-8425: Fix the kustomization procedure to give up after 3 failed retries~~ OCPBUGS-8425: Fix the kustomization procedure not to exit with fatal error Mar 12, 2023

dhellmann reviewed Mar 12, 2023

View reviewed changes

openshift-ci bot assigned dhellmann Mar 12, 2023

openshift-ci bot added the lgtm Indicates that a PR is ready to be merged. label Mar 12, 2023

openshift-merge-robot merged commit 7319d9c into openshift:main Mar 12, 2023

openshift-cherrypick-robot mentioned this pull request Mar 12, 2023

[release-4.13] OCPBUGS-9999: Fix the kustomization procedure not to exit with fatal error #1490

Merged

ggiguash deleted the manifest_retry branch March 13, 2023 04:48

Conversation

ggiguash commented Mar 8, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

openshift-ci-robot commented Mar 8, 2023

Uh oh!

ggiguash commented Mar 8, 2023

Uh oh!

openshift-ci-robot commented Mar 8, 2023

Uh oh!

Uh oh!

Uh oh!

openshift-ci-robot commented Mar 8, 2023

Uh oh!

Uh oh!

Uh oh!

openshift-ci-robot commented Mar 12, 2023

Uh oh!

dhellmann left a comment

Choose a reason for hiding this comment

Uh oh!

openshift-ci bot commented Mar 12, 2023

Uh oh!

openshift-ci bot commented Mar 12, 2023

Uh oh!

openshift-ci-robot commented Mar 12, 2023

Uh oh!

dhellmann commented Mar 12, 2023

Uh oh!

openshift-cherrypick-robot commented Mar 12, 2023

Uh oh!

openshift-cherrypick-robot commented Mar 13, 2023

Uh oh!

ggiguash commented Mar 13, 2023

Uh oh!

openshift-cherrypick-robot commented Mar 13, 2023

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

7 participants

ggiguash commented Mar 8, 2023 •

edited

Loading