[AppStream] Adding QA information to a component

[Matthias Klumpp]

Am Sa., 3. Juli 2021 um 22:05 Uhr schrieb Richard Hughes <hughsient at gmail.com>:
>
> On Sat, 3 Jul 2021 at 20:08, Matthias Klumpp <matthias at tenstral.net> wrote:
> > Maybe you could fire `appstreamcli validate` at the files, just to
> > check that?
>
> I'm not sure I can turn off <checksum> and just rely on <artifact>
> just yet; at the moment we just do "both" so we support old and new
> versions of fwupd. Maybe a few more years!

Appstreamcli shouldn't throw an error yet if `artifact` isn't used...
But it's actually very likely that it will do that in future,
supporting both syntax versions is a bit of a pain and definitely
something that I'll want to go away when AppStream 1.0 is released.

> > I would make the XML look differently though - while freeform
> > "anything goes" key value fields are nice, in the past it always paid
> > off to have more specific tag names and narrowly defined value spaces.
>
> Yes, you're probably right. <value tag="foo">bar<value> is the a{sv}
> of API designs :)

Indeed :-)

> > In AppStream, we usually have one larger tag with children if we
> > expect there to be usually more tags of the same type, which I think
> > applies here (there are exceptions, like the <url/> tag for backwards
> > compatibility).
>
> Yes, agree. At least from a libxmlb point of view, using a parent tag
> enclosure like <testing> means we do one integer compare and a jump,
> compared to N(integer compare and a jump) so it makes a lot of sense.
>
> >   <testing>
> >     <test_result date="2020-07-02">
>
> date! I forgot that bit, 100% agree.
>
> >       <tester>Red Hat QA</tester>
>
> not <name>? It seemed like the "name" of the test_result i..e the bit
> you'd show in the UI.

Ah! I thought the string value was for a person / team that did the
testing, not a name for the specific test that has been performed. For
the latter, `name` is obviously the best choice.

> >       <device>Lenovo Thinkpad P1</device>
> >       <os version="9">rhel</os>
>
> These make sense.
>
> > Questions for this: Is there ever a case where a test has *failed*
> > that should show up here? Or will all listed tests be always
> > successful, so actually be some kind of "it works!" certificates?
>
> I did think about only doing successful tests; hopefully any failures
> would mean the firmware isn't available.

That makes sense. What will the UI do if a test result isn't available
for the specific hardware the user is running, but the firmware itself
is available? Warn about the update? (that would IMHO be a bit
weird...)

> > Since you went for the key/value approach, do you expect there to be a
> > need for freeform key value pairs, like in the custom tag in
> > AppStream? If so, what are the applications for that?
>
> The only other thing that I might need is the version the tester
> upgraded *from* i.e. 1.2.3 to 1.2.4 is much more likely to succeed
> than 0.1.2 to 3.4.5 and also the version of the client that updated
> it. e.g. what I've got so far is:
>
>       <testing>
>         <test_result id="1" date="2021-07-02">
>           <name>LVFS</name>
>           <device>LENOVO ThinkPad X1 Carbon 7th</device>
>           <os version="34" variant="workstation">fedora</os>
>           <custom>
>             <value key="RuntimeVersion(org.freedesktop.fwupd)">1.6.2</value>
>             <value key="VersionOld">1.2.6</value>
>           </custom>
>         </test_result>
>       </testing>
>     </release>
>
> Getting there! Thanks!

Looks sensible! What is the `id` attribute of test_result for?
Ordering? Or just some unique ID for the specific result?

I also have to make one correction: The `platform` attribute that I
mentioned in my previous mail is obviously a property of an *artifact*
and not of a *release*. This means we could either add the testing
block to individual artifacts and not the whole release, or add it to
the release and then duplicate the `platform` info tag. I think the
latter is likely the way to go, as it is the more flexible approach in
case a release doesn't want to use artifact links but still wants to
add the testing information.

That "VersionOld" thing could potentially be generalized (I'd call it
updated_from_version maybe, or previous_version as tag), but the
runtime version piece is very much implementation specific. Either
way, I don't have objections to adding a custom block there, being
able to assign custom metadata to a test result feels like a good
idea.

Cheers,
    Matthias

-- 
I welcome VSRE emails. See http://vsre.info/