REST API: Bulk requests for packages and vulnerabilities

[MarcelBochtler]

We're planning to add support for vulnerablecode into the ORT advisor.
The existing advisor module for Nexus IQ Server is using a single POST request with all PURLs in the body to retrieve vulnerability informations about all the packages, see: https://github.com/oss-review-toolkit/ort/blob/master/clients/nexus-iq/src/main/kotlin/NexusIqService.kt#L109-L110

To be able to use VulnerableCode in a performant way, VulnerableCode's REST API should offer a bulk request API for packages and for vulnerabilities.

[pombredanne]

@MarcelBochtler Excellent request! Let's design how you would like the API to behave.

[MarcelBochtler]

As a GET request would not be feasible to handle all PURLs as a query parameter I'd suggest a POST endpoint containing required package information in its body.

To be consistent to the existing API, I'd suggest to offer this functionality for packages and for vulnerabilites.

Packages

This should return the list of package details incl. the vulnerabilities' reference_ids

For my use case the PURL would be used in the body. E.g.

{
    "packages": [
        {
            "purl": "pkg:pypi/jaraco.logging@1.5"
        },
        {
            "purl": "pkg:maven/org.hamcrest/hamcrest-core@1.3"
        }
    ]
}

As an alternative you could offer this API to use the other package identifiers like type, namespace, name and a version:

{
    "packages": [
        {
            "identifier": {
                "format": "maven",
                "coordinates": {
                    "artifactId": "hamcrest-core",
                    "groupId": "org.hamcrest",
                    "version": "1.3"
                }
            }
        },
        {
            "identifier": {
                "coordinates": {
                    "name": "jaraco.logging",
                    "version": "1.5"
                }
            }
        }
    ]
}

This is based on Sonatype's component details API.
I didn't like the differently named fields for the coordinates, that's why I used the PURL implementation.

Vulnerabilites

{
    "vulnerabilities": [
        {
            "reference_id": "CVE-2020-15250"
        },
        {
            "reference_id": "CVE-2020-8352"
        }
    ]
}

Returning a list of vulnerability details.

[sschuberth]

Shouldn't the response also tell you which package of the bulk request is affected by which vulnerability?

[MarcelBochtler]

The Package model contains lists of resolved_vulnerabilities and unresolved_vulnerabilities both containing vulnerability_ids that can be used to get more details from the Vulnerabilities endpoint.

Edit: And yes the response of the vulnerabilities can contain a list of packages:

[sbs2001]

It makes sense to add a bulk request for packages.

@MarcelBochtler

re. coordinates vs purl string :
coordinates are useful in only the case where you don't have exact idea of what you are looking for. As you suggested, purl string is the way to go in this case.

I also have same concern as @sschuberth has:

Shouldn't the response also tell you which package of the bulk request is affected by which vulnerability?

Here's what I propose:

1. Use request body as suggested

eg

{
    "packages": [
        {
            "purl": "pkg:pypi/jaraco.logging@1.5"
        },
        {
            "purl": "pkg:maven/org.hamcrest/hamcrest-core@1.3"
        }
    ]
}

2. The response should look something like :

{
    "pkg:pypi/jaraco.logging@1.5":{
        
        "resolved_vulnerabilities":[
            {
            "vulnerability_id": "CVE-FOO-BAR",
            "url":"http://127.0.0.1:8000/api/vulnerabilities/21050/"
            }
        ],
        "unresolved_vulnerabilities":[
            {
            "vulnerability_id": "CVE-TAR-BALL",
            "url":"http://127.0.0.1:8000/api/vulnerabilities/210/"
            },
            {
            "vulnerability_id": "CVE-9999-1234",
            "url":"http://127.0.0.1:8000/api/vulnerabilities/211/"
            },
        ]
        
    },

    "pkg:maven/org.hamcrest/hamcrest-core@1.3":{
        "resolved_vulnerabilities":[],
        "unresolved_vulnerabilities":[
            {
            "vulnerability_id": "GHSA-abcd-zxcv",
            "url":"http://127.0.0.1:8000/api/vulnerabilities/2134/"
            },
        ]

    }
}

Thinking about this why not make (1) ie the request body more simple/flatter, like :

[
"pkg:maven/org.hamcrest/hamcrest-core@1.3",
"pkg:pypi/jaraco.logging@1.5"
]

[sbs2001]

@MarcelBochtler will a bulk request for vulnerabilities be useful for ort ? Correct me if I am wrong, ort would be first collecting purls using the Analyser and then querying whether the collected purls are secure or not. So the use case would be along the lines of "I have purl called foo, give me all it's vulnerabilities(if any)" not "I have vulnerability X, give me all packages which are affected by X then I will check whether I use any of them"

[MarcelBochtler]

not "I have vulnerability X, give me all packages which are affected by X then I will check whether I use any of them"

Correct.

But the severity of the CVE, the cvss is required e.g. to use the ORT evaluator for rule evaluation based on this severity.
This value is not contained in Package but only in Vulnerability

That's why a second request would be required to get more details of the found vulnerabilities.

[MarcelBochtler]

Thinking about this why not make (1) ie the request body more simple/flatter, like :
[
"pkg:maven/org.hamcrest/hamcrest-core@1.3",
"pkg:pypi/jaraco.logging@1.5"
]

That would be fine for me. The wrapping is mainly done to support possible future extensions.

[sbs2001]

But the severity of the CVE, the cvss is required

I see, so we can add bulk vulnerability request too.

FYI the cvss thing is oversimplified and I am working on it to capture more types of scores , see #157 (comment) . There are cases where same vulnerability has been provided varying scores by different sources and we want to capture that. On the API side of thing changes will be done to reflect that.

[sbs2001]

I will start working on this after getting #290 and #259 are merged, since those bring some schema changes .