[Piglit] Allow junit to be used for summary generation

[Dylan Baker]

On Fri, Oct 09, 2015 at 10:33:35AM -0700, Dylan Baker wrote:
> On Fri, Oct 09, 2015 at 09:44:01AM -0700, Mark Janes wrote:
> > Jose Fonseca <jfonseca at vmware.com> writes:
> > 
> > > On 09/10/15 01:21, baker.dylan.c at gmail.com wrote:
> > >> This series updates the junit backend to allow it to properly load junit
> > >> and convert it back into piglit's internal representation, thus allowing
> > >> it to be summarized using the piglit summary tools
> > >>
> > >> There is still some work that needs to be done beyond this, most of the
> > >> platform metadata isn't stored yet and restored, but I have a plan for
> > >> that. I have some other refactoring work that I think will make that
> > >> easier, and I'd like to get there before landing that.
> > >>
> > >> This is enough to be able to compare junit and json results using the
> > >> console and html summaries.
> > >>
> > >> There is a caveat here, and that's patch 3. To compare json and junit we
> > >> need to be able to restore the names of the junit tests to *exactly*
> > >> what they were before, and currently we don't have a way to reverse the
> > >> '.' -> '_' conversion. My proposal is to change '.' into '___', which is
> > >> very unlikely in a real test name (though we could change it to almost
> > >> anything that would be unique). This may break some existing setup
> > >> (Mark, I think this will probably break some of our expected fail/crash
> > >> data).
> > >>
> > >
> > > I don't object this, but instead of this brittle testname (de)munge, 
> > > have you considered/tried using an additional XML attribute with 
> > > unmodified piglit test name?  I expect that Jenkins junit parser will 
> > > just outright ignore it.
> > 
> > My recollection is that we found the junit parser to be strict:
> > 
> > https://svn.jenkins-ci.org/trunk/hudson/dtkit/dtkit-format/dtkit-junit-model/src/main/resources/com/thalesgroup/dtkit/junit/model/xsd/junit-4.xsd
> > 
> > >
> > > Jose
> 
> I don't know if we tried, It definately allows some not strictly valid
> junit to be passed to it (IIRC we didn't originally pass the total
> number of tests in, and that is required per the SPEC).
> 
> If it will accept arbitrary tags that would be very nice. We might want
> to guard them behind a --non-strict-junit switch (or inversely provide a
> --strict-junit flag) in case that changes or someone whats to us a
> consumer of junit that is strict.
> 
> I think I'll give it a try and see what happens.

Mark is correct, I added an extra tag and jenkins ignored the result. So
it is at least too strict for this use.
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 473 bytes
Desc: not available
URL: <http://lists.freedesktop.org/archives/piglit/attachments/20151009/c9e7ee52/attachment-0001.sig>