add support to `eessi_container.sh` for using `unionfs` overlay tool by boegel · Pull Request #9 · EESSI/software-layer-scripts

boegel · 2025-06-14T13:53:28Z

Default behavior is left unchanged.

unionfs looks promising, haven't observed the Permission denied issues we've been seeing when creating overlay with fuse-overlays, but it needs to be battle-tested before we jump ship.

Tested with:

./eessi_container.sh --container docker://ghcr.io/eessi/build-node:debian12 --overlay-tool unionfs --access rw

(see also EESSI/filesystem-layer#226)

trz42

Minor suggestions

eessi_container.sh

…iner.sh script Co-authored-by: Thomas Röblitz <trz42@users.noreply.github.com>

…r.sh

…layer-scripts into eessi_container_unionfs

boegel · 2025-06-17T10:00:21Z

.github/workflows/tests_scripts.yml

-          echo 'source /software-layer/scripts/utils.sh' >> ${test_script}
-          echo 'source /software-layer/init/eessi_environment_variables' >> ${test_script}
+          echo 'ls -l /software-layer-scripts/' >> ${test_script}
+          echo 'sed -i "s/__EESSI_VERSION_DEFAULT__/${{matrix.EESSI_VERSION}}/g" init/eessi_defaults


actual fix is here, this looks more complicated than it should because of software-layer -> software-layer-scripts rename...

…AULT__ placeholder in init/eessi_defaults in CI workflow to test load_easybuild_module.sh script

bedroge · 2025-06-24T13:32:27Z

@boegel the CI is still failing because of:

ERROR: /cvmfs/software.eessi.io/versions/__EESSI_VERSION_DEFAULT__/compat/linux/x86_64 does not exist!

Any idea why the EESSI_VERSION_OVERRIDE is not working?

edit: quickly tried to reproduce that locally, but it worked fine for me...

boegel · 2025-06-24T16:41:28Z

@boegel the CI is still failing because of:
ERROR: /cvmfs/software.eessi.io/versions/__EESSI_VERSION_DEFAULT__/compat/linux/x86_64 does not exist!
Any idea why the EESSI_VERSION_OVERRIDE is not working?

edit: quickly tried to reproduce that locally, but it worked fine for me...

I've been trying to figure out why it's not working as it should, but I'm puzzled too...

bedroge · 2025-06-25T07:31:31Z

I tried it again, I think it's because run_in_compat_layer_env.sh does a source ${base_dir}/init/eessi_defaults. That's before you're overriding EESSI_VERSION_OVERRIDE, so then EESSI_VERSION gets set to that placeholder string.

.github/workflows/tests_scripts.yml

Co-authored-by: Bob Dröge <b.e.droge@rug.nl>

… tests

…ipt tests

bedroge · 2025-06-25T12:03:53Z

.github/workflows/tests_scripts.yml

          out="${PWD}/test_create_lmodsitepackage.out"
-          ./eessi_container.sh --mode run --verbose /software-layer/run_in_compat_layer_env.sh /software-layer/test_lmod_sitepackage.sh 2>&1 | tee ${out}
-          for pattern in "^Site Pkg location.*/software-layer/.lmod/SitePackage.lua" "LMOD_SITEPACKAGE_LOCATION.*/software-layer/.lmod/SitePackage.lua"; do
+          ./eessi_container.sh --mode run --verbose /software-layer-scripts/run_in_compat_layer_env.sh /software-layer-scripts/test_lmod_sitepackage.sh 2>&1 | tee ${out}


$EESSI_VERSION_OVERRIDE should also be set for this step

bot/build.sh

bedroge

Lgtm

add support to and for using unionfs overlay tool

9df2fbd

boegel added the enhancement New feature or request label Jun 14, 2025

trz42 requested changes Jun 16, 2025

View reviewed changes

eessi_container.sh Outdated Show resolved Hide resolved

eessi_container.sh Outdated Show resolved Hide resolved

boegel and others added 6 commits June 17, 2025 11:15

don't hardcode software.eessi.io when creating overlay in eessi_conta…

955d0e6

…iner.sh script Co-authored-by: Thomas Röblitz <trz42@users.noreply.github.com>

add test for using '--overlay-tool unionfs' option for eessi_containe…

1351105

…r.sh

add test for using '--overlay-tool unionfs' option for eessi_containe…

dd4cc2a

…r.sh

Merge branch 'main' into eessi_container_unionfs

e3a9eb8

replace EESSI version placeholder in CI workflow for scripts

b421f1d

Merge branch 'eessi_container_unionfs' of github.com:boegel/software-…

badfeba

…layer-scripts into eessi_container_unionfs

boegel commented Jun 17, 2025

View reviewed changes

boegel added 2 commits June 17, 2025 12:04

fix test of using unionfs as overlay-tool

d96fb6f

improve pattern check in test for using unionfs as overlay-tool

c9b81ba

boegel force-pushed the eessi_container_unionfs branch from 3aaa438 to c9b81ba Compare June 17, 2025 18:53

boegel added 3 commits June 17, 2025 21:01

fix checking for use of '--overlay-tool unionfs'

2bbadca

fix replacing of EESSI version placeholder in CI workflow for scripts

8e2e292

set $EESSI_VERSION_OVERRIDE rather than replacing __EESSI_VERSION_DEF…

1675ff9

…AULT__ placeholder in init/eessi_defaults in CI workflow to test load_easybuild_module.sh script

boegel mentioned this pull request Jun 24, 2025

{2023.06}[2023a] Bazel/6.1.0 ml_dtypes/0.3.2 tensorboard/2.15.1 EESSI/software-layer#1131

Merged

boegel added 2 commits June 24, 2025 11:05

Merge branch 'main' into eessi_container_unionfs

1f11ae4

show contents of test script when testing load_easybuild_module.sh

00f3ea2

boegel mentioned this pull request Jun 24, 2025

Create utils function for nvidia-smi check #22

Merged

bedroge reviewed Jun 25, 2025

View reviewed changes

.github/workflows/tests_scripts.yml Show resolved Hide resolved

boegel and others added 3 commits June 25, 2025 10:15

set $EESSI_VERSION_OVERRIDE before calling run_in_compat_layer_env.sh

f9392e6

Co-authored-by: Bob Dröge <b.e.droge@rug.nl>

also set $EESSI_VERSION_OVERRIDE for install_software_layer.sh script…

dd37dfd

… tests

also set $EESSI_VERSION_OVERRIDE for create_directory_tarballs.sh scr…

5f4b557

…ipt tests

bedroge reviewed Jun 25, 2025

View reviewed changes

boegel mentioned this pull request Jun 27, 2025

update Java/11, Java/17, Java/21 wrapper modules to latest patch releases EESSI/software-layer#1102

Merged

bedroge reviewed Jun 27, 2025

View reviewed changes

bot/build.sh Outdated Show resolved Hide resolved

also set for create_lmodsitepackage.py tests

0ec75f1

boegel force-pushed the eessi_container_unionfs branch from a4e7200 to 0ec75f1 Compare June 27, 2025 13:49

bedroge approved these changes Jun 27, 2025

View reviewed changes

bedroge merged commit 3ceef12 into EESSI:main Jun 27, 2025
47 checks passed

boegel deleted the eessi_container_unionfs branch June 27, 2025 14:13

This was referenced Jul 1, 2025

use unionfs overlay tool (+ debian12 build container that includes it) by default #26

Merged

Unable to rebuild on certain architectures EESSI/software-layer#1103

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

add support to `eessi_container.sh` for using `unionfs` overlay tool#9

add support to `eessi_container.sh` for using `unionfs` overlay tool#9
bedroge merged 18 commits intoEESSI:mainfrom
boegel:eessi_container_unionfs

boegel commented Jun 14, 2025

Uh oh!

trz42 left a comment

Uh oh!

Uh oh!

Uh oh!

boegel Jun 17, 2025

Uh oh!

bedroge commented Jun 24, 2025 •

edited

Loading

Uh oh!

boegel commented Jun 24, 2025

Uh oh!

bedroge commented Jun 25, 2025

Uh oh!

Uh oh!

bedroge Jun 25, 2025

Uh oh!

Uh oh!

bedroge left a comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Comments

Conversation

boegel commented Jun 14, 2025

Uh oh!

trz42 left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

boegel Jun 17, 2025

Choose a reason for hiding this comment

Uh oh!

bedroge commented Jun 24, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

boegel commented Jun 24, 2025

Uh oh!

bedroge commented Jun 25, 2025

Uh oh!

Uh oh!

bedroge Jun 25, 2025

Choose a reason for hiding this comment

Uh oh!

Uh oh!

bedroge left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Comments

bedroge commented Jun 24, 2025 •

edited

Loading