WIP 🐛 Workload should still resilient when catalog is deleted #2439

camilamacedo86 · 2026-01-11T05:18:07Z

What This Fixes

Installed extensions now keep working when their catalog becomes unavailable or is deleted.

Scenario	Runtime	On Main	With This PR
Registry offline	Helm	❌ Extension fails	✅ Keeps running
Registry offline	Boxcutter	❌ Extension fails	✅ Keeps running
Catalog deleted + resource deleted	Helm	❌ Resource NOT restored	✅ Auto-restored
Catalog deleted + resource deleted	Boxcutter	❌ Resource NOT restored	✅ Auto-restored
Catalog unavailable + update config	Helm	❌ Update fails	✅ Update works
Catalog unavailable + update config	Boxcutter	❌ Update fails	✅ Update works
Catalog unavailable + upgrade version	Helm	❌ Unclear error	✅ Clear error + keeps running
Catalog unavailable + upgrade version	Boxcutter	❌ Unclear error	✅ Clear error + keeps running

TL'DR: Detailed Scenarios Examples

Scenario 1: Registry Server Goes Offline

Setup: You have Prometheus operator v1.0.0 installed and running

What happens: The container registry where the catalog image is stored becomes unreachable (network issue, registry maintenance, etc.)

On Main Branch ❌

Runtime	What Breaks	Why
Helm	Extension shows "Failed"	Can't contact catalog → reconciliation stops
Boxcutter	Extension shows "Failed"	Can't contact catalog → reconciliation stops

Impact: Prometheus stops monitoring your cluster

With This PR ✅

Runtime	What Works	How
Helm	Extension stays "Installed", keeps running	Falls back to Helm release stored in cluster
Boxcutter	Extension stays "Installed", keeps running	Falls back to ClusterExtensionRevision in cluster

Impact: Prometheus continues monitoring without interruption

## Scenario 2: Catalog Deleted, Resource Gets Deleted

Setup: Prometheus operator installed

What happens:

1. You delete the ClusterCatalog (Any reasons, the catalog was deleted it should not broke current workloads)
2. Someone accidentally runs: kubectl delete configmap prometheus-config

On Main Branch ❌

Runtime	What Happens	Result
Helm	ConfigMap NOT restored	Prometheus breaks, no monitoring
Boxcutter	ConfigMap NOT restored	Prometheus breaks, no monitoring

Why: Catalog deleted → reconciliation stopped → Apply step never runs → resources not maintained

Impact: Manual recovery needed (recreate ConfigMap manually)

With This PR ✅

Runtime	What Happens	How It's Fixed
Helm	ConfigMap automatically restored	Helm.Reconcile() compares cluster to stored release, recreates ConfigMap
Boxcutter	ConfigMap automatically restored	Boxcutter engine SSA from ClusterExtensionRevision, recreates ConfigMap

Impact: Self-healing! Prometheus automatically recovers

Scenario 3: Update Configuration Without Catalog

Setup: Prometheus installed, catalog unavailable

What happens: You want to change CRD upgrade safety enforcement from "Strict" to "None"

On Main Branch ❌

Runtime	What Happens	Error Message
Helm	Update FAILS	"reconciliation failed: catalog not found"
Boxcutter	Update FAILS	"reconciliation failed: catalog not found"

Why: Reconciliation requires catalog lookup

Impact: Can't modify any settings

With This PR ✅

Runtime	What Happens	How
Helm	Update APPLIED	Helm applier updates existing release with new config
Boxcutter	Update APPLIED	Boxcutter updates existing revision with new config

Impact: Configuration management works offline

Scenario 4: Try to Upgrade Version Without Catalog

Setup: Prometheus v1.0.0 installed, catalog unavailable

What happens: You try to upgrade Prometheus to v1.0.1

On Main Branch ❌

Runtime	What Happens	Status Shown
Helm	Upgrade doesn't happen	Generic "Failed"
Boxcutter	Upgrade doesn't happen	Generic "Failed"

Why: Reconciliation stopped

Impact: Unclear why upgrade not happening, no visibility

With This PR ✅

Runtime	What Happens	Status Shown
Helm	Upgrade properly blocked	`Progressing=Retrying: catalog not found`, `Installed=v1.0.0 (True)`
Boxcutter	Upgrade properly blocked	`Progressing=Retrying: catalog not found`, `Installed=v1.0.0 (True)`

Impact:

Clear feedback: "Need catalog to upgrade"
Workload still runs on v1.0.0
Will auto-upgrade when catalog returns (via watch)

netlify · 2026-01-11T05:18:11Z

✅ Deploy Preview for olmv1 ready!

Name	Link
🔨 Latest commit	`865ac9b`
🔍 Latest deploy log	https://app.netlify.com/projects/olmv1/deploys/6964df6f0c1457000898a77d
😎 Deploy Preview	https://deploy-preview-2439--olmv1.netlify.app
📱 Preview on mobile	Toggle QR Code... Use your smartphone camera to open QR code link.

To edit notification comments on pull requests, go to your Netlify project configuration.

openshift-ci · 2026-01-11T05:18:11Z

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by:
Once this PR has been reviewed and has the lgtm label, please assign pedjak for approval. For more information see the Code Review Process.

The full list of commands accepted by this bot can be found here.

Details

Needs approval from an approver in each of these files:

OWNERS

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

Copilot

Pull request overview

This PR adds comprehensive end-to-end tests to verify that installed OLM extensions continue functioning correctly when their source catalog is deleted. The tests cover both standard runtime and experimental Boxcutter runtime scenarios.

Changes:

Added new feature file with 8 scenarios testing catalog deletion resilience
Implemented CatalogIsDeleted function to support catalog deletion in tests
Added step registrations for ClusterExtension update operations

Reviewed changes

Copilot reviewed 2 out of 2 changed files in this pull request and generated 2 comments.

File	Description
test/e2e/steps/steps.go	Adds CatalogIsDeleted function and step registrations for testing catalog deletion and ClusterExtension updates
test/e2e/features/catalog-deletion-resilience.feature	Defines 8 test scenarios covering extension resilience, resource restoration, config changes, version upgrades, and revision behavior when catalog is deleted

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

test/e2e/steps/steps.go

Copilot

Pull request overview

Copilot reviewed 3 out of 3 changed files in this pull request and generated 2 comments.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

test/e2e/steps/steps.go

internal/operator-controller/applier/boxcutter.go

camilamacedo86 · 2026-01-11T06:59:59Z

internal/operator-controller/controllers/clusterextension_reconcile_steps.go

+			l.Info("skipping unpack - using installed bundle content")
+			// imageFS will remain nil - the applier will use the existing installed content
+			return nil, nil
+		}


PROBLEM: Always tries to pull image, even when using installed bundle

camilamacedo86 · 2026-01-11T07:01:35Z

internal/operator-controller/applier/helm.go

+
+	// If contentFS is nil, we're maintaining the current state without catalog access.
+	// In this case, reconcile the existing Helm release if it exists.
+	if contentFS == nil {


Will fail if is not able to get the content.
PROBLEM: Immediately tries to build chart from contentFS
FIX: Reconcile the existing release and watch the release objects to ensure they're maintained

Copilot

Pull request overview

Copilot reviewed 5 out of 5 changed files in this pull request and generated 2 comments.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

test/e2e/features/catalog-deletion-resilience.feature

internal/operator-controller/applier/boxcutter.go

Copilot

Pull request overview

Copilot reviewed 6 out of 6 changed files in this pull request and generated 5 comments.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

internal/operator-controller/applier/boxcutter.go

internal/operator-controller/controllers/clusterextension_reconcile_steps.go

internal/operator-controller/applier/helm.go

internal/operator-controller/controllers/clusterextension_reconcile_steps.go

test/e2e/steps/steps.go

Copilot

Pull request overview

Copilot reviewed 6 out of 6 changed files in this pull request and generated 4 comments.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

internal/operator-controller/controllers/clusterextension_reconcile_steps.go

internal/operator-controller/applier/boxcutter.go

internal/operator-controller/controllers/clusterextension_reconcile_steps.go

test/e2e/steps/steps.go

Copilot

Pull request overview

Copilot reviewed 6 out of 6 changed files in this pull request and generated 5 comments.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

internal/operator-controller/applier/boxcutter.go

test/e2e/steps/steps.go

internal/operator-controller/controllers/clusterextension_reconcile_steps.go

internal/operator-controller/applier/helm.go

Copilot

Pull request overview

Copilot reviewed 7 out of 7 changed files in this pull request and generated 6 comments.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

internal/operator-controller/controllers/clusterextension_reconcile_steps.go

internal/operator-controller/applier/boxcutter.go

test/e2e/steps/steps.go

internal/operator-controller/controllers/clusterextension_reconcile_steps.go

internal/operator-controller/applier/helm.go

Copilot

Pull request overview

Copilot reviewed 8 out of 8 changed files in this pull request and generated 4 comments.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

internal/operator-controller/controllers/clusterextension_controller_test.go

test/e2e/steps/steps.go

internal/operator-controller/controllers/clusterextension_reconcile_steps.go

Copilot

Pull request overview

Copilot reviewed 7 out of 7 changed files in this pull request and generated 1 comment.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

internal/operator-controller/controllers/clusterextension_reconcile_steps.go

Enables installed extensions to continue working when their source catalog becomes unavailable or is deleted. When resolution fails due to catalog unavailability, the operator now continues reconciling with the currently installed bundle instead of failing. Changes: - Resolution falls back to installed bundle when catalog unavailable - Unpacking skipped when maintaining current installed state - Helm and Boxcutter appliers handle nil contentFS gracefully - Version upgrades properly blocked without catalog access This ensures workloads remain stable and operational even when the catalog they were installed from is temporarily unavailable or deleted, while appropriately preventing version changes that require catalog access.

Copilot AI review requested due to automatic review settings January 11, 2026 05:18

openshift-ci bot requested review from ankitathomas and trgeiger January 11, 2026 05:18

Copilot started reviewing on behalf of camilamacedo86 January 11, 2026 05:18 View session

Copilot AI reviewed Jan 11, 2026

View reviewed changes

test/e2e/steps/steps.go Outdated Show resolved Hide resolved

test/e2e/steps/steps.go Outdated Show resolved Hide resolved

camilamacedo86 force-pushed the test-e2e-res branch from d3cbb5a to f31b184 Compare January 11, 2026 05:43

Copilot AI review requested due to automatic review settings January 11, 2026 05:43

Copilot started reviewing on behalf of camilamacedo86 January 11, 2026 05:43 View session

camilamacedo86 changed the title ~~🌱 test: add e2e tests for workload resilience when catalog is deleted~~ WIP 🌱 test: add e2e tests for workload resilience when catalog is deleted Jan 11, 2026

openshift-ci bot added the do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. label Jan 11, 2026

Copilot AI reviewed Jan 11, 2026

View reviewed changes

test/e2e/steps/steps.go Outdated Show resolved Hide resolved

test/e2e/steps/steps.go Outdated Show resolved Hide resolved

camilamacedo86 force-pushed the test-e2e-res branch from f31b184 to dce6d68 Compare January 11, 2026 06:48

camilamacedo86 commented Jan 11, 2026

View reviewed changes

internal/operator-controller/applier/boxcutter.go Outdated Show resolved Hide resolved

camilamacedo86 commented Jan 11, 2026

View reviewed changes

camilamacedo86 force-pushed the test-e2e-res branch from dce6d68 to b15c262 Compare January 11, 2026 07:09

Copilot AI review requested due to automatic review settings January 11, 2026 07:09

Copilot started reviewing on behalf of camilamacedo86 January 11, 2026 07:10 View session

Copilot AI reviewed Jan 11, 2026

View reviewed changes

test/e2e/features/catalog-deletion-resilience.feature Show resolved Hide resolved

internal/operator-controller/applier/boxcutter.go Outdated Show resolved Hide resolved

camilamacedo86 force-pushed the test-e2e-res branch from b15c262 to b1d259e Compare January 11, 2026 07:26

Copilot AI review requested due to automatic review settings January 11, 2026 07:30

camilamacedo86 force-pushed the test-e2e-res branch from b1d259e to c6870c5 Compare January 11, 2026 07:30

Copilot started reviewing on behalf of camilamacedo86 January 11, 2026 07:30 View session

camilamacedo86 changed the title ~~WIP 🌱 test: add e2e tests for workload resilience when catalog is deleted~~ WIP 🐛 Workload should still resilient when catalog is deleted Jan 11, 2026

Copilot AI reviewed Jan 11, 2026

View reviewed changes

camilamacedo86 force-pushed the test-e2e-res branch from c6870c5 to 36e9069 Compare January 11, 2026 08:48

Copilot AI review requested due to automatic review settings January 11, 2026 09:00

camilamacedo86 force-pushed the test-e2e-res branch from 36e9069 to 6799025 Compare January 11, 2026 09:00

Copilot started reviewing on behalf of camilamacedo86 January 11, 2026 09:01 View session

Copilot AI reviewed Jan 11, 2026

View reviewed changes

camilamacedo86 force-pushed the test-e2e-res branch 2 times, most recently from 986e945 to 95f39fb Compare January 11, 2026 09:41

Copilot AI review requested due to automatic review settings January 11, 2026 09:41

Copilot started reviewing on behalf of camilamacedo86 January 11, 2026 09:42 View session

Copilot AI reviewed Jan 11, 2026

View reviewed changes

camilamacedo86 force-pushed the test-e2e-res branch from 95f39fb to 8851183 Compare January 11, 2026 12:03

Copilot AI review requested due to automatic review settings January 11, 2026 18:33

camilamacedo86 force-pushed the test-e2e-res branch from 8851183 to 3f99ac5 Compare January 11, 2026 18:33

Copilot started reviewing on behalf of camilamacedo86 January 11, 2026 18:34 View session

Copilot AI reviewed Jan 11, 2026

View reviewed changes

camilamacedo86 force-pushed the test-e2e-res branch from 3f99ac5 to 20ba323 Compare January 12, 2026 00:38

Copilot AI review requested due to automatic review settings January 12, 2026 09:46

Copilot started reviewing on behalf of camilamacedo86 January 12, 2026 09:47 View session

Copilot AI reviewed Jan 12, 2026

View reviewed changes

camilamacedo86 force-pushed the test-e2e-res branch from 043d689 to 773f156 Compare January 12, 2026 10:57

test: add e2e tests for workload resilience when catalog is deleted

f6ef763

Copilot AI review requested due to automatic review settings January 12, 2026 11:22

camilamacedo86 force-pushed the test-e2e-res branch from 773f156 to 84e6cc6 Compare January 12, 2026 11:22

Copilot started reviewing on behalf of camilamacedo86 January 12, 2026 11:22 View session

Copilot AI reviewed Jan 12, 2026

View reviewed changes

internal/operator-controller/controllers/clusterextension_reconcile_steps.go Outdated Show resolved Hide resolved

camilamacedo86 force-pushed the test-e2e-res branch from 84e6cc6 to 865ac9b Compare January 12, 2026 11:47

WIP 🐛 Workload should still resilient when catalog is deleted #2439

Are you sure you want to change the base?

WIP 🐛 Workload should still resilient when catalog is deleted #2439

Conversation

camilamacedo86 commented Jan 11, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What This Fixes

TL'DR: Detailed Scenarios Examples

Scenario 1: Registry Server Goes Offline

On Main Branch ❌

With This PR ✅

## Scenario 2: Catalog Deleted, Resource Gets Deleted

On Main Branch ❌

With This PR ✅

Scenario 3: Update Configuration Without Catalog

On Main Branch ❌

With This PR ✅

Scenario 4: Try to Upgrade Version Without Catalog

On Main Branch ❌

With This PR ✅

Uh oh!

netlify bot commented Jan 11, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

✅ Deploy Preview for olmv1 ready!

Uh oh!

openshift-ci bot commented Jan 11, 2026

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Uh oh!

Uh oh!

Uh oh!

camilamacedo86 Jan 11, 2026

Choose a reason for hiding this comment

Uh oh!

camilamacedo86 Jan 11, 2026

Choose a reason for hiding this comment

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Copilot AI left a comment

camilamacedo86 commented Jan 11, 2026 •

edited

Loading

netlify bot commented Jan 11, 2026 •

edited

Loading