Workaround that solves permission issues for rebuilds

[bedroge]

Solves #556, tested in #866.

It works around the issue by removing the original directory, then creating a new one with the first-level subdirs (e.g. bin, lib, easybuild, etc), and it makes sure that EB is then run with --try-amend=keeppreviousinstall=True to prevent EB from wiping it again (as it would otherwise run into the same issue again).

[eessi-bot]

Instance eessi-bot-mc-aws is configured to build for:

• architectures: x86_64/generic, x86_64/intel/haswell, x86_64/intel/sapphire_rapids, x86_64/intel/skylake_avx512, x86_64/amd/zen2, x86_64/amd/zen3, aarch64/generic, aarch64/neoverse_n1, aarch64/neoverse_v1
• repositories: eessi.io-2023.06-software, eessi.io-2023.06-compat

[eessi-bot]

Instance eessi-bot-mc-azure is configured to build for:

• architectures: x86_64/amd/zen4
• repositories: eessi.io-2023.06-software, eessi.io-2023.06-compat

[gpu-bot-ugent]

Instance eessi-bot-vsc-ugent is configured to build for:

• architectures: x86_64/amd/zen3
• repositories: eessi-hpc.org-2023.06-software, eessi.io-2023.06-software, eessi.io-2023.06-compat, eessi-hpc.org-2023.06-compat

[boegel]

Interesting how that dances around the issue...

It almost feels like something isn't being synced fast enough, leading to the error we've seen in #556.

By pre-creating directories we essentially allow for more time to "sync", and then the problem doesn't present itself (or at the very least makes it less likely).

Good that we found a workaround for it.
Relying on keeppreviousinstall makes me a bit nervous though, since it was intended for very different purposes, but it should be fine in practice as long as we only have empty directories pre-created.

Can we verify this somehow, maybe in a hook that's triggered when keeppreviousinstall is enabled, as a safeguard?
We definitely don't want to have keeppreviousinstall enabled when the installation directory has anything other than empty subdirectories...

[eessi-bot]

PR merged! Moved [] to /project/def-users/SHARED/trash_bin/EESSI/software-layer/2025.01.22

[eessi-bot]

PR merged! Moved [] to /project/def-users/SHARED/trash_bin/EESSI/software-layer/2025.01.22

[gpu-bot-ugent]

PR merged! Moved [] to /scratch/gent/vo/002/gvo00211/SHARED/trash_bin/EESSI/software-layer/2025.01.22

[boegel]

@bedroge This was merged via #866, but it probably makes sense to put that extra check in place in our hooks?

[bedroge]

@bedroge This was merged via #866, but it probably makes sense to put that extra check in place in our hooks?

Done in #881.