[WIP] update licenses by MartinsNadia · Pull Request #675 · EESSI/software-layer

MartinsNadia · 2024-08-19T14:29:52Z

Open the new one to fix the origin branch

eessi-bot · 2024-08-19T14:29:57Z

Instance eessi-bot-mc-aws is configured to build for:

architectures: x86_64/generic, x86_64/intel/haswell, x86_64/intel/skylake_avx512, x86_64/amd/zen2, x86_64/amd/zen3, aarch64/generic, aarch64/neoverse_n1, aarch64/neoverse_v1
repositories: eessi.io-2023.06-compat, eessi-hpc.org-2023.06-software, eessi-hpc.org-2023.06-compat, eessi.io-2023.06-software

eessi-bot · 2024-08-19T14:29:57Z

Instance eessi-bot-mc-azure is configured to build for:

architectures: x86_64/amd/zen4
repositories: eessi.io-2023.06-compat, eessi-hpc.org-2023.06-compat, eessi-hpc.org-2023.06-software, eessi.io-2023.06-software

MartinsNadia · 2024-08-19T14:34:12Z

WIP

adding a new licenses update script PR 457;
adding draft yml file

ocaisa

Just had an initial look, perhaps you can add the current output of the script in a gist and link to this issue?

ocaisa · 2024-10-15T09:07:22Z

licenses/check_licenses.yml

+        uses: actions/checkout@v3
+
+      - name: Set up Python
+        uses: actions/setup-python@v4


Should use specific version here, see other workflows

ocaisa · 2024-10-15T09:07:49Z

licenses/check_licenses.yml

+          fi 
+
+      - name: Create a PR (if changes detected)
+        uses: peter-evans/create-pull-request@v5


Also needs specific commit

ocaisa · 2024-10-15T09:08:21Z

licenses/check_licenses.yml

+        commit-message: "Auto PR: Update licenses"
+        title: "Auto PR: Update licenses"
+         body: ${{ steps.check_licenses.outputs.patch }}
+         branch: main #fork branch


This is not the correct branch for EESSI

ocaisa · 2025-02-03T11:22:25Z

.github/workflows/check_licenses.yml

+      - name: Create a PR (if changes detected)
+        id: create_pull_request
+        uses: peter-evans/create-pull-request@5e914681df9dc83aa4e4905692ca88beb2f9e91f # v7.0.5
+        if: steps.check_licenses.outputs.patch != ''
+        with: 
+          commit-message: "Auto PR: Update licenses"
+          title: "Auto PR: Update licenses"
+          body: ${{ steps.check_licenses.outputs.patch }}
+          branch: update-licenses-${{ github.run_number }}
+          base: [ "*-software.eessi.io" ]


Doesn't this mean that if a PR adds software, then it will trigger another PR to add the licence of the software to the yaml file? Instead, wouldn't you want to make a suggestion that they add the licence as part of the original PR?

ocaisa · 2025-02-03T11:24:39Z

.github/workflows/check_licenses.yml

+  pull_request:
+    branches:  [ "*-software.eessi.io" ]
+permissions:
+  contents: write # set permissions for writing 


This action is being given a lot of power at a global level, do we really need it to have write permissions? Isn't it enough that we get a notification if it fails?

ocaisa · 2025-02-03T11:26:57Z

.github/workflows/check_licenses.yml

+      - name: Apply patch (if no PR created)
+        if: steps.create_pull_request.outputs.number == '' && steps.check_licenses.outputs.patch != ''        
+        run: |
+          if [ -f license_update.patch ] && [ -s license_update.patch ]; then
+            git apply license_update.patch
+          else
+            echo "No changes to apply"
+          fi
+          git add licenses.json
+          git diff --cached --exit-code || git commit -m "Update licenses.json"
+          git push
+        env:
+          GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}


Rather than auto-apply a patch, can a code suggestion be made (or a PR to the source repo using a token without write permission to EESSI)?

ocaisa · 2025-02-03T11:27:55Z

licenses/check_licenses.yml

@@ -0,0 +1,59 @@
+name: Check and update licenses


Duplicate file?

ocaisa · 2025-02-03T11:29:35Z

.github/workflows/check_licenses.yml

+          python update_licenses.py --source=pypi TensorFlow
+          python update_licenses.py --source=github:easybuilders/easybuild EasyBuild


You are going to need to extract the software name from the easystack file, that is complicated. Perhaps instead you should add easystack (and easyconfig) support to update_licenses.py?

This could also help you decide where to look for licenses (since you have access to source_urls)

The other option is to gather software (and extension) names via Lmod and then check them against the yaml file.

ocaisa · 2025-02-03T11:34:38Z

.github/workflows/check_licenses.yml

+          branch: update-licenses-${{ github.run_number }}
+          base: [ "*-software.eessi.io" ]
+
+      - name: Apply patch (if no PR created)


Why wouldn't the PR be created? Wouldn't that mean the previous step will have failed?

hvelab · 2025-02-27T18:02:34Z

spare thoughts right now:

i kept the retrieval date because the original script did so, but i dont see much sense in keeping it, I would keep the license url instead
there are a lot of not found but i expect them to be less of them as right now some source_urls are still formatted with EB syntax but it my priority to fix it tomorrow
need to fix the "needs manual action" part as its case sensitive right now with the "Other" licenses

ocaisa · 2025-02-28T15:09:17Z

licenses/licenses.json

+    },
+    "ATK/2.38.0-GCCcore-13.2.0": {
+        "License": [
+            "Other"


How did this end up with Other? I see it is GPL v2 at https://gitlab.gnome.org/Archive/atk

ocaisa · 2025-02-28T15:15:38Z

licenses/licenses.json

+    },
+    "Archive-Zip/1.68-GCCcore-12.2.0": {
+        "License": [
+            "Other"


Hmm, some of these are going to need some discussion. The license of this is at https://github.com/redhotpenguin/perl-Archive-Zip?tab=License-1-ov-file#readme and it is GPL1+ or an artistic licence. If we are going to put Other we'll need a link to the licence if possible and possibly another field permission_to_redistribute

Agree, this is the plain response from the ecosyste.ms API: https://repos.ecosyste.ms/api/v1/repositories/lookup?url=https%3A%2F%2Fgithub.com%2Fredhotpenguin%2Fperl-Archive-Zip

Will work today on a "smarter" way to do this and to scrape other sources because there are much more not found licenses than found or other

ocaisa · 2025-02-28T15:18:53Z

licenses/licenses.json

+    },
+    "Arrow/16.1.0-gfbf-2023b": {
+        "License": [
+            "Other"


License for this is https://github.com/apache/arrow/blob/main/LICENSE.txt ("Apache-2.0")

ocaisa · 2025-02-28T15:31:44Z

licenses/licenses.json

+        "Retrieved From": "not found"
+    },
+    "Bazel/6.3.1-GCCcore-12.3.0": {
+        "License": "not found",


Apache 2.0 from https://github.com/bazelbuild/bazel/blob/master/LICENSE

ocaisa · 2025-02-28T15:33:33Z

licenses/licenses.json

+    },
+    "BeautifulSoup/4.12.2-GCCcore-12.3.0": {
+        "License": [
+            "Other"


MIT from https://pypi.org/project/beautifulsoup4/

ocaisa · 2025-02-28T15:35:23Z

licenses/licenses.json

+        "Retrieved From": "https://www.fftw.org"
+    },
+    "FFmpeg/6.0-GCCcore-13.2.0": {
+        "License": "not found",


https://www.ffmpeg.org/legal.html

hvelab · 2025-03-03T16:15:23Z

Updates:

Fixed bug and now finds more licenses
Now shows the real api call from where it got the license
For the "Other" and "not found", does an scraping to go find the LICENSE file -> we need to find a way to retrieve the spdx from there
This needs to be improved for the packages with "Others"
For the totally "not found", shows source_urls and homepages, seems that most of them need to be sanitized or do the scraping from there, add more keywords

hvelab · 2025-03-11T20:21:18Z

finished doing manual checks (while watching some random movie on the TV), we would just need permission for PGPLOT (https://sites.astro.caltech.edu/~tjp/pgplot/#copyright), and for CUDA im not sure if the propietary license allows us to redistribute it
the scraper (aside from update_licenses.json) is good for an initial populate but to cover every single case is very difficult and required much more manual checks than expected, at least is done for now, for all software available in EESSI excluding extensions, which is my next focus, such as enforcing specifying the license from EB files

could try from a hook and if it fails then request the manual action? again, it's very difficult to scrape it from webpages and sometimes requires human interpretation

laraPPr · 2025-06-27T13:05:21Z

@ocaisa should this be closed and moved to https://github.com/EESSI/software-layer-scripts?

update licenses

062b624

ocaisa reviewed Oct 15, 2024

View reviewed changes

MartinsNadia added 2 commits December 9, 2024 20:31

fix commits, branches, identation. Add content and ids

e517a4e

adding licenses yml

1480ae1

ocaisa requested changes Feb 3, 2025

View reviewed changes

needs more refinement but to show the progress for now

bbb8ee9

updated a bit

3355eb2

ocaisa reviewed Feb 28, 2025

View reviewed changes

progress in the license retrieval

2d7826e

torri added 2 commits March 11, 2025 16:39

still some to go but to show progress

3a52f9c

got it!

a267a00

torri added 2 commits April 7, 2025 15:40

cleaning and reworking

e50b25c

cleaning files

f224575

hvelab mentioned this pull request Apr 7, 2025

Software licenses tracking #1011

Closed

		python update_licenses.py --source=pypi TensorFlow
		python update_licenses.py --source=github:easybuilders/easybuild EasyBuild

Conversation

MartinsNadia commented Aug 19, 2024

Uh oh!

eessi-bot bot commented Aug 19, 2024

Uh oh!

eessi-bot bot commented Aug 19, 2024

Uh oh!

MartinsNadia commented Aug 19, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

ocaisa left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

hvelab commented Feb 27, 2025

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

hvelab commented Mar 3, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

hvelab commented Mar 11, 2025

Uh oh!

laraPPr commented Jun 27, 2025

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

MartinsNadia commented Aug 19, 2024 •

edited

Loading

hvelab commented Mar 3, 2025 •

edited

Loading