[Freedesktop-sdk] license-checking script for BuildStream projects

[Adam Jones]

On Tue, 2020-08-25 at 20:22 +0100, Douglas Winship wrote:
> Following on from the previous email, I've put together a basic
> license-checker in python and tested it in a CI Pipeline. I'd be very
> interested to get feedback on the html and json output.
> 
> In particular I'd be interested to get opinions about how to
> implement the blacklist: we're planning to design the license checker
> with a blacklist option, where users can supply a list of blacklisted
> licenses (possibly as regular expressions). If any blacklisted
> licenses are detected, these would be reported in the html and json
> outputs, but I'm not sure what form that ought to take.
> ------------
> 
> Sample html output is here: 
> https://gitlab.com/freedesktop-sdk/freedesktop-sdk/-/jobs/701539612/artifacts/file/buildstream_license_checker_output/url_m

This is super cool!


> anifest/license_check_summary.html
> And Sample json output is here: 
> https://gitlab.com/freedesktop-sdk/freedesktop-sdk/-/jobs/701539612/artifacts/file/buildstream_license_checker_output/url_manifest/license_check_summary.json
> The Pipeline is here: 
> https://gitlab.com/freedesktop-sdk/freedesktop-sdk/-/pipelines/181505791
> And there's an MR here: 
> https://gitlab.com/freedesktop-sdk/freedesktop-sdk/-/merge_requests/3293
> (Note that the MR is still very very much a WIP).
> 
> 
> On 14/08/2020 09:27, Douglas Winship wrote:
> > Hi all. 
> > 
> > 
> > We're looking to develop a basic license-checking script for
> > buildstream 
> > projects, and would appreciate some input, either input on the tool
> > in general, 
> > or input on whether it would be useful for freedesktop-sdk. 
> > 
> > The the user would invoke the script with one line, and the basic
> > process would be: 
> > 
> > * User supplies one or more elements to check 
> > * User also supplies a '--deps' argument (none, run, or all) 
> > * Script invokes bst-show to get a list of relevant dependency
> > elements 
> > * Script checks out the source code for each dependency into a
> > temporary directory 
> > * Script runs license-checking software on the source code,
> > producing a detailed 
> >   output (one output file per dependency element) 
> > * Script takes the raw outputs, and condenses it into a summary. 
> >   (A machine-readable summary, a human-readable summary, or one
> > each.) 
> > 
> > The current plan is to use licensecheck, as the actual license-
> > checking software 
> > (
> > https://metacpan.org/pod/distribution/App-Licensecheck/bin/licensecheck
> > ) 
> > (https://packages.debian.org/buster/licensecheck) 
> > 
> > licensecheck produces output lines for each file in the source
> > code, like this: 
> >     ... 
> >     ./crypto/algif_hash.c: UNKNOWN 
> >     ./crypto/algif_rng.c: BSD (3 clause) GPL 
> >     ./crypto/algif_skcipher.c: UNKNOWN 
> >     ./crypto/ansi_cprng.c: UNKNOWN 
> >     ./crypto/anubis.c: GPL (v2 or later) 
> >     ./crypto/api.c: UNKNOWN 
> >     ... 
> > 
> > The script would then summarize the data into a single summary
> > file, which 
> > would summarize all of the dependencies (summary would be either 
> > machine-readable,  human-readable, or one of each, according to 
> > user-specified arguments). 
> > 
> > The summary format is tbd, but the logical structure of the summary
> > would 
> > look like: 
> >     .... 
> >     - dependency_name: bootstrap/sed.bst 
> >       dependency_fullkey:
> > e52cee70287646e712c427accd3ef9ec533380ae7265177350c58cc3457f10b1 
> >       licensecheck_output: 
> >       - BSD (4 clause) 
> >       - FSF All Permissive 
> >       - GPL 
> >       - GPL (v2 or later) 
> >       - GPL (v3) 
> >       - GPL (v3 or later) 
> >       - ISC GPL (v3 or later) 
> >       - MIT/X11 (BSD like) 
> >       - *No copyright* BSL 
> > 
> >     - dependency_name: bootstrap/stripper.bst 
> >       dependency_fullkey:
> > f2993d8c833ee3aae118c6b2c96d6e7f4dc0cac2947cb0467835f5b288f01175 
> >       licensecheck_output: 
> >       - MIT/X11 (BSD like) 
> >     ... 
> > 
> > (Note, I've used YAML here as an example, because it's more
> > readable for the 
> > purposes of this email, but json may be preferable for the actual
> > output.) 
> > 
> > QUESTIONS: 
> > 
> > A) Would this script be useful for freedesktop-sdk? 
> > B) Would this script be useful in CI? 
> > C) What sort of format would be good for the machine-readable
> > summary? json? YAML? 
> > D) What sort of format would be good for the human-readable
> > summary? markdown? html? 
> > E) What would be a more useful output for freedesktop-sdk: just the
> > summaries? 
> > or should we also include the raw licensecheck output? 
> > 
> > _______________________________________________ 
> > Freedesktop-sdk mailing list 
> > Freedesktop-sdk at lists.freedesktop.org 
> > https://lists.freedesktop.org/mailman/listinfo/freedesktop-sdk 
> 
> _______________________________________________
> Freedesktop-sdk mailing list
> Freedesktop-sdk at lists.freedesktop.org
> https://lists.freedesktop.org/mailman/listinfo/freedesktop-sdk

-- 
--------------------
Codethink Ltd
3rd Floor Dale House
35 Dale Street
Manchester
M1 2HF
United Kingdom

Codethink delivers cutting edge open source design, development and
integration services.

Web: http://codethink.co.uk
Tel: +44 7446 844846
Email: adam.jones at codethink.co.uk