clean up local tmp if it has been saved to a tarball#267
Open
trz42 wants to merge 3 commits intoEESSI:2023.06-software.eessi.iofrom
Open
clean up local tmp if it has been saved to a tarball#267trz42 wants to merge 3 commits intoEESSI:2023.06-software.eessi.iofrom
trz42 wants to merge 3 commits intoEESSI:2023.06-software.eessi.iofrom
Conversation
|
Instance
|
boegel
reviewed
Jun 22, 2023
Collaborator
Author
|
bot: show_config |
Updates by the bot instance
|
Updates by the bot instance
|
|
Instance
|
|
Instance
|
Collaborator
Author
|
bot: show_help |
Updates by the bot instance
|
Updates by the bot instance
|
Collaborator
Author
|
bot: show_config |
Updates by the bot instance
|
Updates by the bot instance
|
|
Instance
|
|
Instance
|
Collaborator
Author
|
bot: show_config |
Updates by the bot instance
|
Updates by the bot instance
|
|
Instance
|
|
Instance
|
trz42
pushed a commit
to trz42/software-layer
that referenced
this pull request
Feb 3, 2024
{2023.06}[foss/2023a] TensorFlow v2.13.0
bedroge
changed the base branch from
2023.06-pilot.eessi-hpc.org
to
2023.06-software.eessi.io
bedroge
reviewed
Apr 16, 2024
| # double-check that at least one tarball of the temporary storage was created | ||
| if check_tmp_tarball ${TARBALL_TMP_BUILD_STEP_DIR} || check_tmp_tarball ${TARBALL_TMP_TARBALL_STEP_DIR}; then | ||
| echo "Removing temporary storage under '${STORAGE}'" | ||
| rm -rf ${STORAGE} |
Collaborator
There was a problem hiding this comment.
If I understand things correctly, $STORAGE refers to the base directory where all jobs create their unique subdirectory? In that case, shouldn't we only remove ${JOB_STORAGE} here, as there may be other (running) jobs on the same machine?
Collaborator
Author
There was a problem hiding this comment.
I think, that's right.
Surprisingly, this ${JOB_STORAGE} is just created (a subdirectory under ${STORAGE}) but it is never used.
We should do two things
- replace uses of
${STORAGE}by${JOB_STORAGE}when runningeessi_container.shand test properly if that still works (maybe a separate PR would be beneficial) - replace uses of
${STORAGE}by${JOB_STORAGE}in this PR here such that only job specific job temporary storage is removed
Collaborator
|
@trz42 will you return to this pr? What is still require? It also seem that these changes should be maid in the repo and not moved to https://github.com/EESSI/software-layer-scripts but please correct me I'm wrong. |
Collaborator
|
Should also retarget this pr if you want to keep it |
Collaborator
|
@trz42 Can you retarget this pr? |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
5 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The
bot/build.shrunseessi_container.shtwice and each run results in a tarball containing the temporary folder. If a job runs on a machine where this temporary folder is not cleaned up automatically, disk usage can pile up leading to failing build jobs.This PR checks if a temporary folder is accessible (
${STORAGE}). If so it removes it if there is any tarball created for it.Improves situation that resulted in creating EESSI/eessi-bot-software-layer#186