Allow to run occ preview:repair in parallel

[MorrisJobke]

Wit this occ preview:repair can be run multiple times in the same instance and will share load between those. It will still process the directory as a whole, but skip locked folders. In this way the processes can run on multiple machines to avoid some bottlenecks like network IO.

The output for one process out of 3 parallel tasks then might look like this:

14:01:09 Migrating previews of file with fileId 4735 …
         Locking "occ preview:repair lock 5317" …
         Skipping because it is locked - another process seems to work on this …
14:01:09 Migrating previews of file with fileId 4736 …
         Locking "occ preview:repair lock 5320" …
         Skipping because it is locked - another process seems to work on this …
14:01:09 Migrating previews of file with fileId 4737 …
         Locking "occ preview:repair lock 5323" …
         Create folder preview/b/2/2/e/d/7/e/4737
         Move preview/4737/256-256-crop.jpg to preview/b/2/2/e/d/7/e/4737
         Move preview/4737/3088-2320-max.jpg to preview/b/2/2/e/d/7/e/4737
         Delete empty folder preview/4737
         Unlocked
         Finished migrating previews of file with fileId 4737 …
14:01:09 Migrating previews of file with fileId 4738 …
         Locking "occ preview:repair lock 5326" …
         Skipping because it is locked - another process seems to work on this …
14:01:09 Migrating previews of file with fileId 4739 …
         Locking "occ preview:repair lock 5329" …
         Skipping because it is locked - another process seems to work on this …
14:01:09 Migrating previews of file with fileId 4740 …
         Locking "occ preview:repair lock 5332" …
         Create folder preview/0/f/3/0/4/e/d/4740
         Move preview/4740/256-256-crop.jpg to preview/0/f/3/0/4/e/d/4740
         Move preview/4740/4096-1627-max.jpg to preview/0/f/3/0/4/e/d/4740
         Delete empty folder preview/4740
         Unlocked
         Finished migrating previews of file with fileId 4740 …
14:01:09 Migrating previews of file with fileId 4741 …
         Locking "occ preview:repair lock 5335" …
         Skipping because it is locked - another process seems to work on this …
14:01:09 Migrating previews of file with fileId 4742 …
         Locking "occ preview:repair lock 5338" …
         Skipping because it is locked - another process seems to work on this …
14:01:09 Migrating previews of file with fileId 4743 …
         Locking "occ preview:repair lock 5341" …
         Create folder preview/d/b/9/e/6/e/e/4743
         Move preview/4743/256-256-crop.jpg to preview/d/b/9/e/6/e/e/4743
         Move preview/4743/4032-3024-max.jpg to preview/d/b/9/e/6/e/e/4743
         Delete empty folder preview/4743
         Unlocked
         Finished migrating previews of file with fileId 4743 …

One can see that two folders are locked and skipped and the third one is processed.

Regarding the timing it seems to be harder to judge, but the overall load is less and the total user time is 9.7s (single process) vs 12.2s (3 processes) and the total system time is 2.3s vs 3.5s (3 processes). It completed after 21s (single process) cs 10 seconds (3 processes). The overhead due to the additional skips can be seen, but it can be distributed across multiple machines or even more processes.

Raw timings:

In one go:

215/215 [============================] 100% 21 secs/21 secs Used Memory: 58.0 MiB
php occ preview:repair -v  9,68s user 2,31s system 48% cpu 24,487 total

In 3 parallel runs:

222/222 [============================] 100% 10 secs/10 secs Used Memory: 54.0 MiB
php occ preview:repair -v  4,02s user 1,13s system 31% cpu 16,236 total
222/222 [============================] 100% 10 secs/10 secs Used Memory: 56.0 MiB
php occ preview:repair -v  4,26s user 1,19s system 13% cpu 41,126 total
222/222 [============================] 100% 9 secs/9 secs Used Memory: 54.0 MiB
php occ preview:repair -v  3,89s user 1,14s system 18% cpu 27,008 total

[kesselb]

I would suggest

        $output->writeln(
            'Will only be printed in verbose mode or higher',
            OutputInterface::VERBOSITY_VERBOSE
        );

for the lock messages. More information: https://symfony.com/doc/current/console/verbosity.html

[MorrisJobke]

I would suggest

        $output->writeln(
            'Will only be printed in verbose mode or higher',
            OutputInterface::VERBOSITY_VERBOSE
        );

for the lock messages. More information: https://symfony.com/doc/current/console/verbosity.html

Done.

[rullzer]

Sure

[rullzer]

@MorrisJobke backport?

[MorrisJobke]

@MorrisJobke backport?

To 20 yes.

[MorrisJobke]

/backport to stable20