[AppStream] Adding QA information to a component

[Matthias Klumpp]

Hi!

Am Fr., 2. Juli 2021 um 21:05 Uhr schrieb Richard Hughes <hughsient at gmail.com>:
>
> Hi all,
>
> Some advice please: The LVFS ships metadata in AppStream format, and I
> think it's 100% comformant now. I'm thinking how to do a new requested
> feature where we can add who has checked and QAd a specific component
> release.

Maybe you could fire `appstreamcli validate` at the files, just to
check that? ;-) It may also make sense to add some additional
validation code to AppStream's validator for things that are
especially relevant to fwupd/LVFS, so people generate good metadata.

> For instance, a specific firmware release might be tested by:
>
>  * The device vendor on Windows 10
>  * The SUSE QA team on Linux x64
>  * The RHEL QA team on Linux ppc64le
>
> Of course I could sprinkle that in the <release> section as
> <x-tested_by> or something, but I thought discussion here might be
> useful. A strawman:
>
> <release>
>  <tested_by date="2020-07-02">
>   <name>Red Hat QA</name>
>   <value key="machine">Lenovo Thinkpad P1</value>
>   <value key="distro">RHEL-9</value>
>   <value key="architecture">ppc64le</value>
>  </tested_by>
> </release>
>
> Other ideas very welcome. Thanks!

It's getting really hard to define what AppStream actually is, but
"metadata to describe software" (as generic as it gets...), is
probably actually fine. In other words, I see no harm in adding a
feature like this.
I would make the XML look differently though - while freeform
"anything goes" key value fields are nice, in the past it always paid
off to have more specific tag names and narrowly defined value spaces.
This makes strict validation possible, is a lot more extensible (which
so far we always wanted to do at some point) and makes the XML easily
machine-interpretable.

AppStream already has a platform definition, which is an attribute for
binary releases. E.g. you can have `<release type="binary"
platform="x86_64-linux-gnu" />` - what is missing is distribution
information. If we are going to add this feature, I would want to make
it work for firmware as well as any other software, as this actually
feels like a useful thing to add to other QA workflows as well.
In AppStream, we usually have one larger tag with children if we
expect there to be usually more tags of the same type, which I think
applies here (there are exceptions, like the <url/> tag for backwards
compatibility).
Quick idea, which still needs refining:

```
 <release type="binary" platform="any-linux-gnu">
  <testing>
    <test_result date="2020-07-02">
      <tester>Red Hat QA</tester>
      <device>Lenovo Thinkpad P1</device>
      <os version="9">rhel</os>
      <platform>ppc64le-linux-gnu</platform>
    </test_result>
  </testing>
 </release>
```

The `tester` field would be a free-form text of the person or team or
CI who conducted the test, `device` would be some canonical device
name (no way for AppStream to verify that, hopefully people will use
consistent naming and we won't get "Lenovo Thinkpad P1" and "Lenovo(R)
Thinkpad P1" and "Lenovo Thinkpad P 1"...). The `os` tag would have a
version attribute and the OS ID as value, as defined in
/etc/os-release for the respective distribution. This would allow some
logic like "was tested on all releases lower than RHEL 9" in future,
if we ever need that.
The `platform` would be an AppStream-style normalized GNU triplet, and
be generally optional, because if the release itself already defined a
platform, the release having been tested for a different one makes no
sense.

Questions for this: Is there ever a case where a test has *failed*
that should show up here? Or will all listed tests be always
successful, so actually be some kind of "it works!" certificates?
Since you went for the key/value approach, do you expect there to be a
need for freeform key value pairs, like in the custom tag in
AppStream? If so, what are the applications for that?
As usual, all naming choices are also subject for debate ;-)

Cheers,
    Matthias




-- 
I welcome VSRE emails. See http://vsre.info/