[Freedesktop-sdk] license-checking script for BuildStream projects

[Douglas Winship]

Following on from the previous email, I've put together a basic 
license-checker in python and tested it in a CI Pipeline. I'd be very 
interested to get feedback on the html and json output.

In particular I'd be interested to get opinions about how to implement 
the blacklist: we're planning to design the license checker with a 
blacklist option, where users can supply a list of blacklisted licenses 
(possibly as regular expressions). If any blacklisted licenses are 
detected, these would be reported in the html and json outputs, but I'm 
not sure what form that ought to take.

------------

  * Sample html output is here:
    https://gitlab.com/freedesktop-sdk/freedesktop-sdk/-/jobs/701539612/artifacts/file/buildstream_license_checker_output/url_manifest/license_check_summary.html
  * And Sample json output is here:
    https://gitlab.com/freedesktop-sdk/freedesktop-sdk/-/jobs/701539612/artifacts/file/buildstream_license_checker_output/url_manifest/license_check_summary.json
  * The Pipeline is here:
    https://gitlab.com/freedesktop-sdk/freedesktop-sdk/-/pipelines/181505791
  * And there's an MR here:
    https://gitlab.com/freedesktop-sdk/freedesktop-sdk/-/merge_requests/3293

(Note that the MR is still very very much a WIP).


On 14/08/2020 09:27, Douglas Winship wrote:
> Hi all.
>
>
> We're looking to develop a basic license-checking script for buildstream
> projects, and would appreciate some input, either input on the tool in 
> general,
> or input on whether it would be useful for freedesktop-sdk.
>
> The the user would invoke the script with one line, and the basic 
> process would be:
>
> * User supplies one or more elements to check
> * User also supplies a '--deps' argument (none, run, or all)
> * Script invokes bst-show to get a list of relevant dependency elements
> * Script checks out the source code for each dependency into a 
> temporary directory
> * Script runs license-checking software on the source code, producing 
> a detailed
>   output (one output file per dependency element)
> * Script takes the raw outputs, and condenses it into a summary.
>   (A machine-readable summary, a human-readable summary, or one each.)
>
> The current plan is to use licensecheck, as the actual 
> license-checking software
> (https://metacpan.org/pod/distribution/App-Licensecheck/bin/licensecheck)
> (https://packages.debian.org/buster/licensecheck)
>
> licensecheck produces output lines for each file in the source code, 
> like this:
>     ...
>     ./crypto/algif_hash.c: UNKNOWN
>     ./crypto/algif_rng.c: BSD (3 clause) GPL
>     ./crypto/algif_skcipher.c: UNKNOWN
>     ./crypto/ansi_cprng.c: UNKNOWN
>     ./crypto/anubis.c: GPL (v2 or later)
>     ./crypto/api.c: UNKNOWN
>     ...
>
> The script would then summarize the data into a single summary file, 
> which
> would summarize all of the dependencies (summary would be either
> machine-readable,  human-readable, or one of each, according to
> user-specified arguments).
>
> The summary format is tbd, but the logical structure of the summary would
> look like:
>     ....
>     - dependency_name: bootstrap/sed.bst
>       dependency_fullkey: 
> e52cee70287646e712c427accd3ef9ec533380ae7265177350c58cc3457f10b1
>       licensecheck_output:
>       - BSD (4 clause)
>       - FSF All Permissive
>       - GPL
>       - GPL (v2 or later)
>       - GPL (v3)
>       - GPL (v3 or later)
>       - ISC GPL (v3 or later)
>       - MIT/X11 (BSD like)
>       - *No copyright* BSL
>
>     - dependency_name: bootstrap/stripper.bst
>       dependency_fullkey: 
> f2993d8c833ee3aae118c6b2c96d6e7f4dc0cac2947cb0467835f5b288f01175
>       licensecheck_output:
>       - MIT/X11 (BSD like)
>     ...
>
> (Note, I've used YAML here as an example, because it's more readable 
> for the
> purposes of this email, but json may be preferable for the actual 
> output.)
>
> QUESTIONS:
>
> A) Would this script be useful for freedesktop-sdk?
> B) Would this script be useful in CI?
> C) What sort of format would be good for the machine-readable summary? 
> json? YAML?
> D) What sort of format would be good for the human-readable summary? 
> markdown? html?
> E) What would be a more useful output for freedesktop-sdk: just the 
> summaries?
> or should we also include the raw licensecheck output?
>
> _______________________________________________
> Freedesktop-sdk mailing list
> Freedesktop-sdk at lists.freedesktop.org
> https://lists.freedesktop.org/mailman/listinfo/freedesktop-sdk
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://lists.freedesktop.org/archives/freedesktop-sdk/attachments/20200825/83120399/attachment.htm>