[PATCH i-g-t v3 0/7] benchmarks/kms_fb_stress: Test all possible combinations of formats

[Pekka Paalanen]

On Thu, 21 Mar 2024 12:27:20 -0300
Arthur Grillo <arthurgrillo at riseup.net> wrote:

> On 20/03/24 12:42, Pekka Paalanen wrote:
> > On Thu, 07 Mar 2024 16:41:02 -0300
> > Arthur Grillo <arthurgrillo at riseup.net> wrote:
> >   
> >> After some discussion[1], I changed how to tackle the problem.
> >>
> >> With the upcoming addition of YUV format to VKMS[2] it would be great to
> >> test the their performance. So, test all possible combinations of
> >> formats.  
> > 
> > Hi Arthur,
> > 
> > what do you gain by testing all possible combinations of formats on
> > planes?
> > 
> > I would have thought that if you can pick and choose a format, then you
> > don't need to test combinations. Each format alone (but as plane source
> > and writeback destination separately) would exercise all the VKMS code
> > there is for that format, given that VKMS uses the intermediate pixel
> > format for all blending etc. operations.  
> 
> Yeah, that does make a lot more sense. I think I was overcomplicating
> things, sorry XD. About "but as plane source and writeback destination
> separately", I didn't quite understand that part, could you talk more
> about it?

If you think of VKMS code, you have separate functions for reading and
writing each pixel format. Both would be good to test. Read functions
are used on KMS plane FBs, and write functions are used on writeback
connector FBs.

Here we are talking about performance benchmarking, but some other
tests should be doing correctness testing, too.

> > 
> > 
> > What will the standard benchmark results printout look like?
> > 
> > I imagine it should be a simple table that people could put in their
> > commit message, something like this:
> > 
> > kms_fb_stress test results, plane source:
> > 
> >             before    after    difference
> >   ARGB8888   3.6 s    3.7 s       +2.7 %
> >   ABGR8888   3.6 s    3.8 s       +5.6 %
> >   YUV420    15.5 s    9.9 s        -36 %
> >   YUV410         -    8.8 s          -
> > 
> > and another table for writeback destination, if that's interesting.
> > 
> > But that requires both before and after runs, so one would need to save
> > results in a file, and use a tool to generate the table. Well, maybe
> > something to consider in the future. One could also get even fancier
> > with statistical analysis, but I guess this would be plenty already.  
> 
> Interesting ideas, maybe, for now, just the format and the time it took
> run would suffice? What do you think?

Yes, that's exactly what a single benchmark run should print. Maybe for
all supported formats (but not format combinations ;-).

That's all the benchmark itself can do. Then a developer would run it
before and after changes and save the results into files, and use some
script to turn the files into that table. So the benchmark printout
should be easily machine-readable.

The reason I proposed the original benchmark to have all those KMS
planes with different pixel formats was that I was trying to cram
everything into a single benchmark result number. It's an ok first
step, but once you get more pixel formats, it's both too much and not
enough. Ideally there would be two kinds of benchmark scenarios:

- pixel format oriented, that mainly measures the performance of
  converting each pixel format to or from the VKMS intermediate pixel
  format, and

- blending and color operation oriented, that does lots of blending and
  runs all the color operations (like GAMMA LUT or CTM, once VKMS
  supports those).

For the former, you only really need a single KMS plane. Or maybe two
at most, to be able to use odd positions and odd sizes and maybe also
for a partially off-screen plane. This benchmark would be printing a
time for each pixel format.

The latter would use several overlapping planes to stress the blending.
This benchmark could probably do with just a couple different
configurations, and does not need to iterate over pixel formats. So
this benchmark is what you mostly already have.

Because VKMS converts everything to its internal intermediate pixel
format, the blending performance is largely independent of the pixel
formats. You can test pixel formats more efficiently by minimizing
blending, and blending tests don't need to use every possible pixel
format.


Thanks,
pq
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 833 bytes
Desc: OpenPGP digital signature
URL: <https://lists.freedesktop.org/archives/igt-dev/attachments/20240322/f863c56d/attachment.sig>