🐛 Bug Report: very slow migration

👟 Reproduction steps

When upgrading my current AppWrite to the newest version, the migration takes multiple days.

👍 Expected behavior

Faster migration regardless of record size

👎 Actual Behavior

When upgrading my current AppWrite to the newest version, the migration takes multiple days. I have ~30 million records, but I feel like that shouldn't be impacting the migration since the database itself isn't changing, just the AppWrite code. Is there any solutions on how I can make this process faster, or is there something on AppWrite's end that you can do to optimize large migrations?

The platform has to be down during a migration, and with a 4 day estimation, that's pretty extreme for a scheduled downtime.

🎲 Appwrite version

Version 1.3.x

💻 Operating system

Linux

🧱 Your Environment

No response

👀 Have you spent some time to check if this issue has been raised before?

• I checked and didn't find similar issue

🏢 Have you read the Code of Conduct?

• I have read the Code of Conduct

[joeyouss]

Hi!
Thank you for bringing this up, yes we are aware of this happening and we have a WIP PR for this #5876

[stnguyen90]

@AidenY69, what versions are you upgrading between?

I feel like that shouldn't be impacting the migration since the database itself isn't changing, just the AppWrite code

When running migrations, the database schema and documents are updated and Appwrite must iterate over everything.

[tripolskypetr]

Right now I am using these PowerShell scripts because I need to export data into JSON format. That method is simple and guarantee the data to be readable, combinable from deferent sources and storable in Git
https://github.com/react-declarative/appwrite-backup-tool

[tripolskypetr]

@joeyouss I think JSON format export feature will be quite useful for AppWrite CLI

for (const collection of collections) {
  console.log(`Making backup of ${collection.$id}`);
  await fs.mkdir(`backup/databases/${database.$id}/${collection.$id}`, {
    recursive: true,
  });
  for await (const document of listDocuments(database.$id, collection.$id)) {
    console.log(
      `Writing ${document.$id} from ${collection.$id} from ${database.$id}`
    );
    await fs.writeFile(
      `backup/databases/${database.$id}/${collection.$id}/${document.$id}.json`,
      JSON.stringify(document, null, 2)
    );
  }
}

[DH-555]

@tripolskypetr Is this still relevant?

[tripolskypetr]

It not even worked all that time. Do you got manual tester in the team? Or just using fake unit tests so nobody know will it suit on the production

@DH-555

[DH-555]

@tripolskypetr There is a team of volunteer QA testers from the community that test features and releases manually prior to production releases.
Also, the team tests manually and implements automated tests covering both the Appwrite codebase, Open Runtimes, console and the Utopia adapters that Appwrite uses to ensure the correct functionality even prior to the QA releases.

I think that in previous versions, it was needed to iterate in some cases over all the documents inside a collection since the database scheme changed, so it could take a lot of time.
Probably the most recent migrations don't need that changes.

Anyways I think there should be a feature to migrate with 0 downtime so you can have the migration running in the background so it doesn't affects your app or cause disruption.

[DH-555]

I suggested this:
#8278

So that way migrations don't block the entire instance meanwhile they are being processed.
The migration procedure is complex and could lead into inconsistent data if not handle properly so at least that proposed solution could work temporarily.

[DH-555]

In the case of a migration that requires changes to a extremely large database and has an active amount of users (0 downtime is a must), it's possible to use a load balance in front of the instance, create a copy of the instance or the database, migrate all the data, and then once the old data was migrated, migrate the newest data and keep it synced (the data that was added meanwhile the migrations were being processed) with functions for example.
That way, once both replicas are synced, you switch to the one that is already migrated and delete the old one.

I understand it's not an easy approach and that should be somewhat automatic or in a more efficient way.