perf(preview): Split preview data to new table

[CarlSchwan]

• Resolves: Split previews out of the filecache table #54513

Summary

The new oc_previews table is optimized for storing previews and should decrease significantly the space taken by previews in the filecache table.

This attend to reuse the IObjectStore abstraction over S3/Swift/Azure but currently only support one single bucket configuration.

TODO

• Implement multibucket configuration
• Implement multibucket configuration where previews are stored in different bucket than the original files
• Add migration for local storage
• Add migration for object store storage
• Add migration for multibucket object store storage
• Make migration efficient
• Adapt/Add unit tests
• Re-thing BackgroundCleanupJob and other jobs and commands

TODO moved to seperate MR

• Sharding: we want to do the sharding on the file_id table. Likely could be done with a tool like citus or vitess. Need some research
• Server side encryption: currently we don't use SSE on the previews. If we end up implementing SSE for previews, we should use the same key file as the original files and add a seperate IPreviewStorage implementation using the IEncryptionModule. I already added an encrypted bool to the oc_previews table in preparation as it would be needed anyway for the migration

Checklist

• Code is properly formatted
• Sign-off message is added to all commits
• Tests (unit, integration, api and/or acceptance) are included
• Screenshots before/after for front-end changes Should not contains any visual changes ;)
• Documentation (manuals or wiki) has been updated or is not required
• Backports requested where applicable (ex: critical bugfixes)

[Altahrim]

Quick thoughts on the new table

[CarlSchwan]

@icewind1991 some questions for you :)

• With allow configuring multiple object store backends #52786 there is now the possibility of using multiple backends per user. The current mapping is done using user configs (oc_preferences) and previews are still stored in the 'root' object store. Should we add a column to the preview table for the backend used (e.g. 'default', 'new', 'old', 'root', ...) to allow an easier migration of the previews in the future? Or do we keeping using only the 'root' backend for previews?

• Multi bucket support. Currently either all the previews are stored on the root bucket or with an opt-in flag can be spread out in 256. I keep the exact same mapping in the new system, does it make sense? Or do we want to use the opportunity of having a breaking change to change that?

• Sharding: I added a storage_id field to the column with sharding in mind. I didn't configure yet anything else or test it. Should I had it to SHARD_PRESETS in Connection.php?

• Generally how do you feel about the direction this is going?

[github-actions]

Possible performance regression detected

Show Output

An unhandled exception has been thrown:
TypeError: array_map(): Argument #2 ($array) must be of type array, null given in /home/runner/work/server/server/apps/profiler/lib/Command/Compare.php:35
Stack trace:
#0 /home/runner/work/server/server/apps/profiler/lib/Command/Compare.php(35): array_map()
#1 /home/runner/work/server/server/3rdparty/symfony/console/Command/Command.php(326): OCAProfilerCommandCompare->execute()
#2 /home/runner/work/server/server/core/Command/Base.php(218): SymfonyComponentConsoleCommandCommand->run()
#3 /home/runner/work/server/server/3rdparty/symfony/console/Application.php(1078): OCCoreCommandBase->run()
#4 /home/runner/work/server/server/3rdparty/symfony/console/Application.php(324): SymfonyComponentConsoleApplication->doRunCommand()
#5 /home/runner/work/server/server/3rdparty/symfony/console/Application.php(175): SymfonyComponentConsoleApplication->doRun()
#6 /home/runner/work/server/server/lib/private/Console/Application.php(187): SymfonyComponentConsoleApplication->run()
#7 /home/runner/work/server/server/console.php(91): OCConsoleApplication->run()
#8 /home/runner/work/server/server/occ(33): require_once('...')
#9 {main}

[github-actions]

Possible performance regression detected

Show Output

An unhandled exception has been thrown:
TypeError: array_map(): Argument #2 ($array) must be of type array, null given in /home/runner/work/server/server/apps/profiler/lib/Command/Compare.php:35
Stack trace:
#0 /home/runner/work/server/server/apps/profiler/lib/Command/Compare.php(35): array_map()
#1 /home/runner/work/server/server/3rdparty/symfony/console/Command/Command.php(326): OCAProfilerCommandCompare->execute()
#2 /home/runner/work/server/server/core/Command/Base.php(218): SymfonyComponentConsoleCommandCommand->run()
#3 /home/runner/work/server/server/3rdparty/symfony/console/Application.php(1078): OCCoreCommandBase->run()
#4 /home/runner/work/server/server/3rdparty/symfony/console/Application.php(324): SymfonyComponentConsoleApplication->doRunCommand()
#5 /home/runner/work/server/server/3rdparty/symfony/console/Application.php(175): SymfonyComponentConsoleApplication->doRun()
#6 /home/runner/work/server/server/lib/private/Console/Application.php(187): SymfonyComponentConsoleApplication->run()
#7 /home/runner/work/server/server/console.php(91): OCConsoleApplication->run()
#8 /home/runner/work/server/server/occ(33): require_once('...')
#9 {main}

[artonge]

- Do we have a way to garbage collect orphan previews? Was answered during our call.

[szaimen]

🎉🎉🎉🎉🎉

[ChristophWurst]

Uneducated guess, but an this smells like it will be the primary column to look up rows. Add an index?

[ChristophWurst]

The mapper confirms that an index on file_id+version would be beneficial :)

[AndyScherzinger]

^ @CarlSchwan