Fix foreign key constraints and the mysql procedure that is used #7381

harikrishna-patnala · 2023-03-30T06:58:18Z

Description

This PR fixes #7370 where same foreign key names are used for multiple tables which causes issues with different MySQL versions.

I've updated the procedure to be more precise and altered the problematic foreign keys in upgrade files. This way we can log the errors and resume the upgrade.

Also removed the mysql statements related to foreign key procedure creation and usage from 4.18.0 schema files (those are moved to 4.18.1 java upgrade files) to avoid errors caused in #7370 for certain MySQL versions.

Types of changes

Breaking change (fix or feature that would cause existing functionality to change)
New feature (non-breaking change which adds functionality)
Bug fix (non-breaking change which fixes an issue)
Enhancement (improves an existing feature and functionality)
Cleanup (Code refactoring and cleanup, that may add test cases)

Bug Severity

Screenshots (if appropriate):

How Has This Been Tested?

harikrishna-patnala · 2023-03-30T07:04:31Z

@blueorangutan package

blueorangutan · 2023-03-30T07:05:04Z

@harikrishna-patnala a Jenkins job has been kicked to build packages. It will be bundled with

SystemVM template(s). I'll keep you posted as I make progress.

engine/schema/src/main/resources/META-INF/db/schema-41810to41900.sql

blueorangutan · 2023-03-30T07:46:04Z

Packaging result: ✔️ el7 ✔️ el8 ✔️ el9 ✔️ debian ✔️ suse15. SL-JID 5809

weizhouapache

@harikrishna-patnala
we also need to remove the codes in 41720to41800.sql
otherwise, the users who use mysql 5.6/5.7 will still fail to upgrade to 4.18.1.0 due to same error in #7358

engine/schema/src/main/resources/META-INF/db/schema-41810to41900.sql

codecov · 2023-03-30T08:45:23Z

Codecov Report

Merging #7381 (58c87c2) into 4.18 (03910c2) will increase coverage by 0.00%.
The diff coverage is 0.00%.

@@            Coverage Diff            @@
##               4.18    #7381   +/-   ##
=========================================
  Coverage     12.98%   12.99%           
- Complexity     8984     8996   +12     
=========================================
  Files          2716     2716           
  Lines        256329   256390   +61     
  Branches      39974    39984   +10     
=========================================
+ Hits          33275    33307   +32     
- Misses       218892   218915   +23     
- Partials       4162     4168    +6

Impacted Files	Coverage Δ
...va/com/cloud/upgrade/dao/DatabaseAccessObject.java	`81.25% <0.00%> (-16.25%)`	⬇️
...ain/java/com/cloud/upgrade/dao/DbUpgradeUtils.java	`80.00% <0.00%> (-12.31%)`	⬇️
...ava/com/cloud/upgrade/dao/Upgrade41800to41810.java	`6.66% <0.00%> (-5.34%)`	⬇️

... and 5 files with indirect coverage changes

📣 We’re building smart automated test selection to slash your CI/CD build times. Learn more

github-advanced-security · 2023-03-30T09:04:17Z

You have successfully added a new SonarCloud configuration ``. As part of the setup process, we have scanned this repository and found no existing alerts. In the future, you will see all code scanning alerts on the repository Security tab.

harikrishna-patnala · 2023-03-30T11:13:38Z

@harikrishna-patnala we also need to remove the codes in 41720to41800.sql otherwise, the users who use mysql 5.6/5.7 will still fail to upgrade to 4.18.1.0 due to same error in #7358

Yes, I'll remove it 41720to41800.sql file

harikrishna-patnala · 2023-03-31T02:38:01Z

@blueorangutan package

blueorangutan · 2023-03-31T02:43:03Z

@harikrishna-patnala a Jenkins job has been kicked to build packages. It will be bundled with

SystemVM template(s). I'll keep you posted as I make progress.

harikrishna-patnala · 2023-03-31T05:21:23Z

@blueorangutan package

blueorangutan · 2023-03-31T05:22:03Z

@harikrishna-patnala a Jenkins job has been kicked to build packages. It will be bundled with KVM, XenServer and VMware SystemVM templates. I'll keep you posted as I make progress.

weizhouapache · 2023-03-31T06:22:44Z

@harikrishna-patnala
I think we do not necessarily need a mysql procedure.

Refer to the following code block

    public void dropKey(Connection conn, String tableName, String key, boolean isForeignKey)
    {
        String alter_sql_str;
        if (isForeignKey) {
            alter_sql_str = "ALTER TABLE " + tableName + " DROP FOREIGN KEY " + key;
        } else {
            alter_sql_str = "ALTER TABLE " + tableName + " DROP KEY " + key;
        }
        try(PreparedStatement pstmt = conn.prepareStatement(alter_sql_str);)
        {
            pstmt.executeUpdate();
            s_logger.debug("Key " + key + " is dropped successfully from the table " + tableName);
        } catch (SQLException e) {
            s_logger.debug("Ignored SQL Exception when trying to drop " + (isForeignKey ? "foreign " : "") + "key " + key + " on table "  + tableName + " exception: " + e.getMessage());

        }
    }

blueorangutan · 2023-06-10T08:23:31Z

[SF] Trillian test result (tid-6717)
Environment: kvm-rocky8 (x2), Advanced Networking with Mgmt server r8
Total time taken: 49651 seconds
Marvin logs: https://github.com/blueorangutan/acs-prs/releases/download/trillian/pr7381-t6717-kvm-rocky8.zip
Smoke tests completed. 107 look OK, 1 have errors, 0 did not run
Only failed and skipped tests results shown below:

Test	Result	Time (s)	Test File
test_08_upgrade_kubernetes_ha_cluster	`Failure`	598.91	test_kubernetes_clusters.py

DaanHoogland

clgtm, some malicious upgrade tests required, ... @vladimirpetrov ;)

vladimirpetrov · 2023-06-12T21:50:09Z

@blueorangutan package

blueorangutan · 2023-06-12T21:52:03Z

@vladimirpetrov a [LL] Jenkins job has been kicked to build packages. It will be bundled with KVM, XenServer and VMware SystemVM templates. I'll keep you posted as I make progress.

blueorangutan · 2023-06-12T22:58:33Z

Packaging result [LL]: ✔️ el7 ✔️ el8 ✔️ el9 ✔️ debian ✔️ suse15. SL-JID 6134

weizhouapache

code lgtm

need manual testing

engine/schema/src/main/resources/META-INF/db/schema-41800to41810.sql

DaanHoogland · 2023-06-20T11:45:13Z

code lgtm

need manual testing

@weizhouapache , with manual testing you mean upgrade tests, right?

weizhouapache · 2023-06-20T11:48:47Z

code lgtm
need manual testing

@weizhouapache , with manual testing you mean upgrade tests, right?

@DaanHoogland yes, probably better with mysql 5.6/5.7 which did not work when upgrade to 4.18.0.0

DaanHoogland · 2023-06-20T12:04:03Z

code lgtm
need manual testing

@weizhouapache , with manual testing you mean upgrade tests, right?

@DaanHoogland yes, probably better with mysql 5.6/5.7 which did not work when upgrade to 4.18.0.0

cc @vladimirpetrov

harikrishna-patnala · 2023-06-22T07:36:36Z

@blueorangutan package

blueorangutan · 2023-06-22T07:38:03Z

@harikrishna-patnala a [SF] Jenkins job has been kicked to build packages. It will be bundled with KVM, XenServer and VMware SystemVM templates. I'll keep you posted as I make progress.

blueorangutan · 2023-06-22T08:32:10Z

Packaging result [SF]: ✔️ el7 ✔️ el8 ✔️ el9 ✔️ debian ✔️ suse15. SL-JID 6308

rohityadavcloud · 2023-06-23T11:10:11Z

@blueorangutan test

blueorangutan · 2023-06-23T11:12:03Z

@rohityadavcloud a [SF] Trillian-Jenkins test job (centos7 mgmt + kvm-centos7) has been kicked to run smoke tests

harikrishna-patnala · 2023-06-27T08:17:45Z

@blueorangutan test

blueorangutan · 2023-06-27T08:18:03Z

@harikrishna-patnala a [SF] Trillian-Jenkins test job (centos7 mgmt + kvm-centos7) has been kicked to run smoke tests

harikrishna-patnala · 2023-06-28T06:34:04Z

@blueorangutan package

blueorangutan · 2023-06-28T06:36:03Z

@harikrishna-patnala a [SF] Jenkins job has been kicked to build packages. It will be bundled with KVM, XenServer and VMware SystemVM templates. I'll keep you posted as I make progress.

blueorangutan · 2023-06-28T07:28:25Z

Packaging result [SF]: ✔️ el7 ✔️ el8 ✔️ el9 ✔️ debian ✔️ suse15. SL-JID 6359

harikrishna-patnala · 2023-06-30T02:11:56Z

@blueorangutan test

blueorangutan · 2023-06-30T02:14:05Z

@harikrishna-patnala a [SF] Trillian-Jenkins test job (centos7 mgmt + kvm-centos7) has been kicked to run smoke tests

blueorangutan · 2023-06-30T13:53:46Z

[SF] Trillian test result (tid-6909)
Environment: kvm-centos7 (x2), Advanced Networking with Mgmt server 7
Total time taken: 40485 seconds
Marvin logs: https://github.com/blueorangutan/acs-prs/releases/download/trillian/pr7381-t6909-kvm-centos7.zip
Smoke tests completed. 107 look OK, 1 have errors, 0 did not run
Only failed and skipped tests results shown below:

Test	Result	Time (s)	Test File
test_01_migrate_VM_and_root_volume	`Error`	79.78	test_vm_life_cycle.py
test_02_migrate_VM_with_two_data_disks	`Error`	53.44	test_vm_life_cycle.py

vladimirpetrov

LGTM based on manual testing. I tested some upgrades from 4.14, 4.15, 4.16 and 4.17 using KVM and VMware and found no issues.

harikrishna-patnala force-pushed the FixForeignKey branch from e61520b to 22948a3 Compare March 30, 2023 06:59

harikrishna-patnala changed the title ~~Fix foreign key~~ Fix foreign key constraints and the mysql procedure that is being used Mar 30, 2023

harikrishna-patnala changed the title ~~Fix foreign key constraints and the mysql procedure that is being used~~ Fix foreign key constraints and the mysql procedure that is used Mar 30, 2023

harikrishna-patnala requested a review from weizhouapache March 30, 2023 07:01

harikrishna-patnala force-pushed the FixForeignKey branch from 22948a3 to 9292d3f Compare March 30, 2023 07:04

harikrishna-patnala added this to the 4.19.0.0 milestone Mar 30, 2023

weizhouapache reviewed Mar 30, 2023

View reviewed changes

engine/schema/src/main/resources/META-INF/db/schema-41810to41900.sql Outdated Show resolved Hide resolved

harikrishna-patnala force-pushed the FixForeignKey branch from 9292d3f to 601a360 Compare March 30, 2023 07:21

harikrishna-patnala modified the milestones: 4.19.0.0, 4.18.1.0 Mar 30, 2023

harikrishna-patnala changed the base branch from main to 4.18 March 30, 2023 07:22

boring-cyborg bot added component:agent component:api component:build component:management-server component:orchestration component:storage labels Mar 30, 2023

harikrishna-patnala force-pushed the FixForeignKey branch from 601a360 to 43b7eb1 Compare March 30, 2023 07:24

weizhouapache reviewed Mar 30, 2023

View reviewed changes

engine/schema/src/main/resources/META-INF/db/schema-41810to41900.sql Outdated Show resolved Hide resolved

DaanHoogland approved these changes Jun 12, 2023

View reviewed changes

DaanHoogland assigned vladimirpetrov Jun 12, 2023

weizhouapache approved these changes Jun 20, 2023

View reviewed changes

weizhouapache reviewed Jun 20, 2023

View reviewed changes

engine/schema/src/main/resources/META-INF/db/schema-41800to41810.sql Outdated Show resolved Hide resolved

Removed unused procedure

58c87c2

weizhouapache added the Severity:BLOCKER label Jun 22, 2023

apache deleted a comment from blueorangutan Jun 30, 2023

vladimirpetrov approved these changes Jul 5, 2023

View reviewed changes

DaanHoogland merged commit 0cbe770 into apache:4.18 Jul 6, 2023

Fix foreign key constraints and the mysql procedure that is used #7381

Fix foreign key constraints and the mysql procedure that is used #7381

Uh oh!

Conversation

harikrishna-patnala commented Mar 30, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Description

Types of changes

Bug Severity

Screenshots (if appropriate):

How Has This Been Tested?

Uh oh!

harikrishna-patnala commented Mar 30, 2023

Uh oh!

blueorangutan commented Mar 30, 2023

Uh oh!

Uh oh!

blueorangutan commented Mar 30, 2023

Uh oh!

weizhouapache left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

codecov bot commented Mar 30, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

github-advanced-security bot commented Mar 30, 2023

Uh oh!

harikrishna-patnala commented Mar 30, 2023

Uh oh!

harikrishna-patnala commented Mar 31, 2023

Uh oh!

blueorangutan commented Mar 31, 2023

Uh oh!

harikrishna-patnala commented Mar 31, 2023

Uh oh!

blueorangutan commented Mar 31, 2023

Uh oh!

weizhouapache commented Mar 31, 2023

Uh oh!

blueorangutan commented Jun 10, 2023

Uh oh!

DaanHoogland left a comment

Choose a reason for hiding this comment

Uh oh!

vladimirpetrov commented Jun 12, 2023

Uh oh!

blueorangutan commented Jun 12, 2023

Uh oh!

blueorangutan commented Jun 12, 2023

Uh oh!

weizhouapache left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

DaanHoogland commented Jun 20, 2023

Uh oh!

weizhouapache commented Jun 20, 2023

Uh oh!

DaanHoogland commented Jun 20, 2023

Uh oh!

harikrishna-patnala commented Jun 22, 2023

Uh oh!

blueorangutan commented Jun 22, 2023

Uh oh!

blueorangutan commented Jun 22, 2023

Uh oh!

rohityadavcloud commented Jun 23, 2023

Uh oh!

blueorangutan commented Jun 23, 2023

Uh oh!

harikrishna-patnala commented Jun 27, 2023

Uh oh!

blueorangutan commented Jun 27, 2023

Uh oh!

harikrishna-patnala commented Jun 28, 2023

Uh oh!

blueorangutan commented Jun 28, 2023

Uh oh!

harikrishna-patnala commented Mar 30, 2023 •

edited

Loading

codecov bot commented Mar 30, 2023 •

edited

Loading