[Freedesktop-sdk] License blacklisting [Was: license-checking script for BuildStream projects]

[Tristan Van Berkom]

Hi Douglas,

On Thu, 2020-08-27 at 15:49 +0100, Douglas Winship wrote:
> Hi Tristan
> 
> Thanks for your comments. Your main suggestion sounds good, but I don't 
> follow all of the logic.

Let me try to clarify.

> For clarity, I'll call my existing approach to license checking an 
> "external" approach, since it's an external script that isn't part of
> BuildStream itself, whereas your proposal would be an "internal" approach.

This "external" vs "internal" approach you are pointing out is missing
an important point.

This is only "external" inasmuch as it is implemented on top of the
buildstream CLI.

Since our intention is to upstream this to be a part of the BuildStream
ecosystem, and maintained under the BuildStream umbrella, it becomes
very "internal" as well, especially that maintaining it as such is a
strong statement that:

  This is _the_ way to have a fairly plug and play approach to
  automatically scanning your project sources for licenses.

> The internal approach sounds like an excellent proposal, and as I 
> understand it you're suggesting that both approaches have valid use
> cases, and both approaches should be developed.

I don't think both approaches to *blacklisting* should be developed at
all, and I don't think that this approach to blacklisting licenses is a
particularly useful one.

That does not mean that this automatic license checker does not provide
good value on it's own either (maybe this nuance was lost in my
previous mail, sorry if I was not more clear on that).

If the BuildStream ecosystem is going to provide an approach for
asserting your build has no incompatible licenses for your
distribution, it should only provide one, correct way.

> (With the external approach
> presumably being developed first, 
> since the internal approach is currently blocked.) This makes sense, but if I 
> understand you correctly you're also saying:

Only the plausible SPDX enhancement is blocked, or, could even be
developped in different ways.

E.g. BuildElements could be extended in projects, to optionally copy
SPDX files into %{install-root}/spdx/%{project-name}/%{element-name}/,
and this data could be consumed in artifact form without depending on
element sources.

That said, the SPDX automation is really just an enhancement which
provides some automation in the off chance that both:

 * An upstream module happens to include an SPDX file (I'm not sure
   just how wide spread these are throughout the ecosystem yet).

 * You, as a distribution, happen to trust that the upstream SPDX file
   is well maintained.

As the distributor of the binary, you still have the responsibility of
knowing the content you distribute, and the blame cannot be passed
upstream to a maintainer for providing an SPDX file which is out of
date.

In the end, the SPDX thing is nice to have, but the main ability to
assert license validity is more critically important.

> 1. The internal approach will implement blacklist processing.
> 2. ...and therefore the external approach shouldn't.
> 
> That's the part that confuses me. I don't understand how the one thing 
> implies the other.

A few things:

  (A) The internal approach is not an automated license checking tool,
      the automated license checking tool which provides summaries of
      licenses however is very good input to help guide developers
      in figuring out what license a module is distributed under.

  (B) The internal approach almost invariably requires manual
      annotations in the project data, those annotations are a result
      of a necessary audit performed by the developer (this audit
      is greatly assisted by the license checking script).

> If only one of the approaches is worth using, then we should only 
> develop that approach and not develop the other. But if both approaches
> are worth having, then we should develop both approaches properly. There's no
> reason to deliberately limit the functionality of one approach, by removing
> an obvious and reasonable feature.

These are not two approaches to the same problem, they solve separate
problems.

> Identifying blacklist violations does seem like an obvious feature for 
> any license-checking approach. So far, most of the people I've spoken to
> about the external script seems to have assumed it will be used this way;
> to monitor which licenses apply to the code in their BuildStream project,
> and make sure that doesn't include any licenses that ought to be avoided.
> That effectively means checking against a blacklist.

I think we're convoluting multiple pieces of a puzzle into the same
problem here, ultimately these separate pieces come together to
eventually provide an experience which should allow blacklisting.

One key thing to keep in mind, is that there is no blacklisting
happening with prior audits and manual attribution of licenses to
BuildStream elements, to presume/advertise otherwise would be
irresponsible on our part.

> The external script produces two main summary outputs: a json output for 
> machine processing, and an html output for human reading. The main use case I 
> see for the human-readable output is for users to skim through it, looking for 
> anything out of place or surprising. This is a perfect place for blacklist processing: 
> violations could be highlighted at the top of the page, where they would be visible in one 
> glance.

> Likewise, we're suggesting that the script should be used in CI for 
> projects like freedesktop-sdk. If the script includes blacklist processing, then it 
> can cause CI pipelines to fail when a violation is detected; that's a useful feature. 
> Without blacklist processing, the script would just create output artifacts that 
> people probably won't remember to look at. I don't see what value that adds to CI.

It exactly adds the reports, which are time/resource consuming to
produce on your own, and much easier to parse than checking out the raw
source code for each module and examining them manually.

> Honestly, I think blacklist processing could be the most valuable 
> feature of the external script. If we don't include it, then I'm not sure I understand 
> what the external script is supposed to be for.

Again, this is useful whenever an audit of the release is performed, it
is helpful when the BuildStream project maintainer needs to add
annotations to elements.

Cheers,
    -Tristan